Jul 27 20:51:34.111502 ip-10-0-131-89 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:51:34.507297 ip-10-0-131-89 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:34.507297 ip-10-0-131-89 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:51:34.507297 ip-10-0-131-89 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:34.507297 ip-10-0-131-89 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:51:34.507297 ip-10-0-131-89 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:34.509414 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.509294    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:51:34.515379 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515363    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.515379 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515380    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515385    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515388    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515392    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515395    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515398    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515400    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515403    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515406    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515408    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515411    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515418    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515421    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515424    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515426    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515429    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515431    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515434    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515437    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515439    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.515445 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515442    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515445    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515447    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515450    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515454    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515458    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515462    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515465    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515467    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515470    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515472    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515475    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515477    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515480    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515482    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515485    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515488    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515490    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515492    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.515921 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515495    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515497    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515500    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515502    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515505    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515507    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515510    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515512    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515515    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515517    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515519    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515522    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515524    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515527    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515529    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515532    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515535    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515538    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515540    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515543    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.516406 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515545    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515547    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515550    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515553    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515555    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515557    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515560    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515563    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515565    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515567    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515570    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515578    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515581    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515583    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515586    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515589    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515591    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515594    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515596    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515599    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.516933 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515601    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515605    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515609    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515613    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515616    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515618    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515620    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515624    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515626    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515629    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515632    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515635    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515637    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515640    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515642    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515645    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515649    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515652    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.515655    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.517442 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516409    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516416    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516420    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516423    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516426    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516429    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516432    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516435    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516437    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516440    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516443    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516446    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516448    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516451    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516453    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516456    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516459    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516461    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516466    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.517898 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516469    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516474    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516477    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516480    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516483    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516486    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516488    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516491    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516493    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516496    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516499    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516501    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516503    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516506    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516508    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516511    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516513    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516516    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516518    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516529    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.518475 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516531    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516534    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516536    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516540    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516544    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516547    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516550    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516552    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516555    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516558    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516561    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516563    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516566    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516569    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516572    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516575    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516577    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516580    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.518975 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516582    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516585    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516588    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516590    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516593    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516595    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516598    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516601    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516603    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516606    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516608    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516611    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516615    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516617    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516620    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516623    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516626    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516628    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516631    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516633    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.519431 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516636    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516638    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516642    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516645    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516649    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516651    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516654    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516658    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516661    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516663    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516666    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516669    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516672    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516674    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516677    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516679    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516682    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516684    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516686    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516689    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.519906 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516692    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.516694    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517133    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517139    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517143    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517145    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517149    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517152    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517155    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517157    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517160    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517163    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517165    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517168    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517170    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517173    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517177    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517180    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517184    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.520411 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517187    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517191    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517193    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517196    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517200    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517202    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517206    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517208    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517227    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517230    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517232    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517235    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517238    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517240    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517244    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517248    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517252    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517254    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517257    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.520858 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517259    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517263    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517267    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517270    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517272    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517275    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517278    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517280    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517283    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517285    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517287    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517290    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517297    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517300    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517303    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517305    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517308    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517311    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517313    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.521333 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517316    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517318    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517321    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517324    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517327    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517330    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517332    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517335    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517337    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517340    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517342    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517345    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517348    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517351    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517353    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517356    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517359    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517362    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517364    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517367    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.521825 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517369    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517371    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517374    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517376    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517379    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517382    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517385    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517388    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517390    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517393    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517395    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517398    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517400    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517402    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517405    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517407    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517410    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517412    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517415    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517417    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.522319 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517420    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517422    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517425    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.517428    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517495    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517503    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517510    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517514    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517519    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517522    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517527    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517531    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517534    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517538    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517541    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517544    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517547    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517551    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517554    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517557    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517559    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517562    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517566    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517569    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:51:34.522803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517572    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517575    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517578    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517582    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517585    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517588    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517591    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517594    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517597    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517600    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517603    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517606    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517611    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517614    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517617    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517620    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517623    2580 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517627    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517632    2580 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517635    2580 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517638    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517641    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517644    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517651    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:51:34.523381 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517654    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517657    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517660    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517663    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517667    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517670    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517673    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517675    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517678    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517681    2580 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517685    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517687    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517690    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517693    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517696    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517699    2580 flags.go:64] FLAG: --help="false"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517702    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517705    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517708    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517711    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517716    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517719    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517722    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:51:34.523953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517724    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517727    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517730    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517733    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517736    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517739    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517742    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517745    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517747    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517750    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517753    2580 flags.go:64] FLAG: --lock-file=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517756    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517759    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517762    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517768    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517771    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517773    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517777    2580 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517779    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517782    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517785    2580 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517788    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517793    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517796    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517800    2580 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:51:34.524563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517803    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517806    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517809    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517812    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517816    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517819    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517822    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517829    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517832    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517835    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517838    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517841    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517847    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517850    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517853    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517856    2580 flags.go:64] FLAG: --port="10250"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517859    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517862    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04643828b65f7c46d"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517874    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517877    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517880    2580 flags.go:64] FLAG: --register-node="true"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517883    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517886    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517889    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:51:34.525236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517892    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517894    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517898    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517901    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517904    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517907    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517910    2580 flags.go:64] FLAG: --runonce="false"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517913    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517916    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517919    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517922    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517925    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517928    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517931    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517934    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517937    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517940    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517943    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517945    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517948    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517952    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517954    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517960    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517963    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517966    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:51:34.525805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517970    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517974    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517977    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517979    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517982    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517985    2580 flags.go:64] FLAG: --v="2"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517989    2580 flags.go:64] FLAG: --version="false"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517993    2580 flags.go:64] FLAG: --vmodule=""
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.517997    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.518000    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518125    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518128    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518132    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518135    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518138    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518141    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518143    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518146    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518148    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518151    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518154    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518157    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.526410 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518159    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518163    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518167    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518170    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518173    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518175    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518179    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518182    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518184    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518187    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518189    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518193    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518196    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518198    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518201    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518204    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518207    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518225    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518228    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518231    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.526944 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518233    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518236    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518238    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518241    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518244    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518246    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518249    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518252    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518254    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518258    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518262    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518265    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518269    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518273    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518276    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518278    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518281    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518283    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518286    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518289    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.527492 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518292    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518294    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518297    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518301    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518304    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518306    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518309    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518311    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518314    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518317    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518319    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518322    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518325    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518327    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518330    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518332    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518335    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518338    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518340    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518343    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.527977 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518346    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518348    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518351    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518354    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518356    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518359    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518361    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518364    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518367    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518369    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518372    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518374    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518377    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518380    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518382    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518386    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518388    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518391    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518394    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.528485 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518396    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518399    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518401    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518404    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518406    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518409    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518411    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.518414    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.519056    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.525710    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.525727    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.525747    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.525753    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525811    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525816    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.528956 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525819    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525822    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525825    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525827    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525830    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525833    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525837    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525841    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525844    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525847    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525850    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525853    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525855    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525858    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525861    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525864    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525867    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525869    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525872    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.529354 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525875    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525878    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525882    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525887    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525890    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525892    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525895    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525898    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525902    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525905    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525907    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525910    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525912    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525915    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525918    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525920    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525923    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525925    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525928    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.529809 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525930    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525932    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525935    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525938    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525940    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525942    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525945    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525948    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525951    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525953    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525956    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525958    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525961    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525964    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525968    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525971    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525973    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525976    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525979    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.530277 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525982    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525985    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525988    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525991    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525993    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525996    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.525999    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526002    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526004    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526007    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526009    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526012    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526014    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526017    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526019    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526022    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526024    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526027    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526029    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526032    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.530743 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526034    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526037    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526039    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526042    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526044    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526046    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526049    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526051    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526054    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526056    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526059    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526061    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526064    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526067    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526070    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526073    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526076    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526079    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526081    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.531341 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526084    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.526089    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526203    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526207    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526232    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526237    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526241    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526244    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526247    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526250    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526252    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526255    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526258    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526261    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526263    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:34.531842 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526266    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526268    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526271    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526274    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526276    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526279    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526281    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526284    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526286    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526289    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526292    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526295    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526298    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526301    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526304    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526307    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526309    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526312    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526315    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526317    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:34.532234 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526320    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526323    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526325    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526328    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526330    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526333    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526335    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526338    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526340    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526342    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526345    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526347    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526350    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526352    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526354    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526357    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526359    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526362    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526364    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526367    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:34.532715 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526369    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526371    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526374    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526376    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526379    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526382    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526385    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526387    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526390    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526392    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526395    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526397    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526400    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526402    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526405    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526407    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526410    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526412    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526415    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526418    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:34.533189 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526420    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526423    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526426    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526428    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526431    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526436    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526438    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526441    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526443    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526447    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526450    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526454    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526457    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526461    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526463    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526466    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526469    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526472    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526474    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:34.533734 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526477    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526479    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526491    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526494    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526497    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526499    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:34.526502    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.526507    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.526621    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.528654    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.529910    2580 server.go:1019] "Starting client certificate rotation"
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.530018    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.530050    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:34.534654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.534407    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:34.538259 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.538238    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:34.550645 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.550618    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:51:34.552757 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.552744    2580 log.go:25] "Validated CRI v1 image API"
Jul 27 20:51:34.553557 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.553541    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:51:34.556614 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.556588    2580 fs.go:135] Filesystem UUIDs: map[49cf8b61-f958-49ab-97bd-635aa0b2f031:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f09331d0-6339-4b6f-8faf-7803a0b6a0c9:/dev/nvme0n1p4]
Jul 27 20:51:34.556712 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.556612    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:51:34.561354 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.561333    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:34.561659 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.561546    2580 manager.go:216] Machine: {Timestamp:2026-07-27 20:51:34.560438035 +0000 UTC m=+0.334370445 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3047398 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bed6dd80b8a6e2217ebf1533f8fde SystemUUID:ec2bed6d-d80b-8a6e-2217-ebf1533f8fde BootID:4fd8d2a8-7fff-4b2e-9a4b-6e9440a0d7e4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:78:43:a8:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:78:43:a8:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:34:88:22:14:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:51:34.561762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.561660    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:51:34.561812 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.561790    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:51:34.564869 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.564837    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:51:34.565049 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.564873    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:51:34.565647 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.565633    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:51:34.565703 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.565651    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:51:34.565703 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.565662    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:34.566513 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.566501    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:34.567422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.567409    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:51:34.569495 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569481    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:34.569682 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569670    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:51:34.569756 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569713    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:51:34.569756 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569730    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:51:34.569756 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569751    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:51:34.569877 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569765    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:51:34.569877 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.569778    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:51:34.570174 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.570159    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:34.570253 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.570183    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:34.572560 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.572544    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:51:34.574363 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.574350    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:51:34.574407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.574381    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:51:34.576009 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.575975    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:51:34.576009 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576012    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576021    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576029    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576041    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576055    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576067    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576074    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576084    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576092    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576104    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:51:34.576151 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.576112    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:51:34.577068 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.577050    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:51:34.577165 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.577152    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:51:34.580562 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.580540    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:51:34.580643 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.580608    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:51:34.580692 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.580646    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:51:34.581196 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.581185    2580 server.go:1268] "Started kubelet"
Jul 27 20:51:34.581371 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.581305    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:51:34.581431 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.581375    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:51:34.581431 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.581333    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:51:34.582121 ip-10-0-131-89 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:51:34.583078 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.583057    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:51:34.583697 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.583678    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:51:34.586924 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.586027    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-89.ec2.internal.18c640bfbeb097a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-89.ec2.internal,UID:ip-10-0-131-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-89.ec2.internal,},FirstTimestamp:2026-07-27 20:51:34.581155748 +0000 UTC m=+0.355088218,LastTimestamp:2026-07-27 20:51:34.581155748 +0000 UTC m=+0.355088218,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-89.ec2.internal,}"
Jul 27 20:51:34.589045 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.589011    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:51:34.590400 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.590385    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:51:34.590468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.590401    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:34.590534 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.590479    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.591001 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.590987    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:51:34.591084 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591064    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:51:34.591084 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591082    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:51:34.591156 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591123    2580 factory.go:55] Registering systemd factory
Jul 27 20:51:34.591156 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591136    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:51:34.591250 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591228    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:51:34.591250 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591239    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:51:34.591665 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591616    2580 factory.go:153] Registering CRI-O factory
Jul 27 20:51:34.591665 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591632    2580 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:51:34.591850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591682    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:51:34.591850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591706    2580 factory.go:103] Registering Raw factory
Jul 27 20:51:34.591850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591722    2580 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:51:34.591983 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.591876    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8j74h"
Jul 27 20:51:34.592686 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.592669    2580 manager.go:349] Starting recovery of all containers
Jul 27 20:51:34.592983 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.592948    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 20:51:34.593387 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.593362    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 20:51:34.600595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.600570    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8j74h"
Jul 27 20:51:34.603838 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.603819    2580 manager.go:354] Recovery completed
Jul 27 20:51:34.608506 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.608490    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.611853 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.611835    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.611939 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.611867    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.611939 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.611878    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.612425 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.612410    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:51:34.612425 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.612423    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:51:34.612425 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.612425    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.612585 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.612437    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:34.613055 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613041    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.613102 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613067    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.613102 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613077    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.613906 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613894    2580 policy_none.go:49] "None policy: Start"
Jul 27 20:51:34.613958 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613909    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:51:34.613958 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.613920    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:51:34.614914 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.614904    2580 policy_none.go:47] "Start"
Jul 27 20:51:34.651488 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.651290    2580 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:51:34.651617 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.651523    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:51:34.651617 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.651536    2580 server.go:87] "Starting device plugin registration server"
Jul 27 20:51:34.651804 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.651791    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:51:34.651848 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.651809    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:51:34.651939 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.651916    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:51:34.652051 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.652009    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:51:34.652051 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.652019    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:51:34.652537 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.652512    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:51:34.652611 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.652572    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.752458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.752426    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.754551 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.754532    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.754662 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.754569    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.754662 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.754584    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.754662 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.754620    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.762477 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.762427    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.762477 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.762450    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-89.ec2.internal\": node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.778781 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.778750    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.797138 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.797099    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:51:34.798370 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.798355    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:51:34.798427 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.798392    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:51:34.798427 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.798415    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:51:34.798487 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.798447    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:51:34.802484 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.802468    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:34.879726 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.879676    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.899189 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.899157    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"]
Jul 27 20:51:34.899328 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.899268    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.900651 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.900634    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.900749 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.900671    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.900749 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.900685    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.901989 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.901974    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.902094 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.902141 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902109    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.902733 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902716    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.902826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902742    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.902826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902756    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.902826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902771    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.902826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902788    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.902826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.902797    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.903898 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.903884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.903898 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.903906    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:34.904634 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.904620    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:34.904728 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.904648    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:34.904728 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.904661    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:34.933970 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.933948    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.ec2.internal\" not found" node="ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.938269 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.938253    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-89.ec2.internal\" not found" node="ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.980693 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:34.980649    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:34.994105 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.994079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cbe27aeb4e99555881d264eb730f693-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"9cbe27aeb4e99555881d264eb730f693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.994257 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.994107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:34.994257 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:34.994128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.081160 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.081075    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:35.094488 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cbe27aeb4e99555881d264eb730f693-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"9cbe27aeb4e99555881d264eb730f693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.094488 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.094644 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.094644 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.094644 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/336dbe90a42e70f001ef4a93975ef496-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal\" (UID: \"336dbe90a42e70f001ef4a93975ef496\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.094644 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.094572    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9cbe27aeb4e99555881d264eb730f693-config\") pod \"kube-apiserver-proxy-ip-10-0-131-89.ec2.internal\" (UID: \"9cbe27aeb4e99555881d264eb730f693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.181810 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.181779    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:35.236310 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.236283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.241103 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.241081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.282566 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.282530    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:35.389316 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.389199    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:35.493650 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.493619    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-89.ec2.internal\" not found"
Jul 27 20:51:35.496019 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.496002    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:35.531421 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.531382    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:51:35.531872 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.531580    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:35.531872 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.531609    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:35.569933 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.569894    2580 apiserver.go:52] "Watching apiserver"
Jul 27 20:51:35.590678 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.590650    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.590843 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.590656    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:35.594604 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.594570    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:51:35.595166 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.595142    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b82kq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx","openshift-cluster-node-tuning-operator/tuned-t77v8","openshift-image-registry/node-ca-jnmb6","openshift-network-diagnostics/network-check-target-pxcw2","openshift-ovn-kubernetes/ovnkube-node-xk7hl","openshift-dns/node-resolver-52wdx","openshift-multus/multus-additional-cni-plugins-dwpt7","openshift-multus/multus-lwpmh","openshift-multus/network-metrics-daemon-fd7cv","openshift-network-operator/iptables-alerter-4qglz","kube-system/global-pull-secret-syncer-m582w"]
Jul 27 20:51:35.596983 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.596963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.598150 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.598132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.599482 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.599465    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:51:35.599559 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.599548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:51:35.599638 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.599620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q74xz\""
Jul 27 20:51:35.599762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.599745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.600520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.600488    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.600611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.600533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:51:35.600783 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.600766    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.600859 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.600842    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-scm48\""
Jul 27 20:51:35.601066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.601050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.602027 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.602011    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.602418 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.602405    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:35.602507 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.602486    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:35.602980 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.602964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v66dj\""
Jul 27 20:51:35.603262 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.603246    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.603609 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.603592    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.604142 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604119    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:35.604241 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604172    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.604241 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604201    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"
Jul 27 20:51:35.604241 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nsqjl\""
Jul 27 20:51:35.604241 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604175    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:51:35.604732 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604718    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.604956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.604936    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.605199 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.605155    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:46:35 +0000 UTC" deadline="2028-03-12 21:01:13.856856628 +0000 UTC"
Jul 27 20:51:35.605311 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.605199    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14256h9m38.251661094s"
Jul 27 20:51:35.606221 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.606202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.606606 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.606580    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:35.606803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.606785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.606803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.606795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:51:35.606952 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.606796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:51:35.607908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.607717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.607908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.607777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.607908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.607783    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:51:35.607908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.607792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:51:35.607908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.607833    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.608632 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.608611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lw4z9\""
Jul 27 20:51:35.608725 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.608621    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.608781 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.608737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m6xzk\""
Jul 27 20:51:35.608859 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.608845    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:51:35.608922 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.608907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:51:35.609475 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.609458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.609594 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.609543    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:35.610212 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.609742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-67m2m\""
Jul 27 20:51:35.610212 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.609849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.610212 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.610037    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:51:35.610212 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.610042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.610602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.610585    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:51:35.610908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.610894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s2wrx\""
Jul 27 20:51:35.611597 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.611576    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.613372 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.613339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal"]
Jul 27 20:51:35.613541 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.613528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.613625 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.613604    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:35.614019 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.614001    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:35.614609 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.614586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jvrwr\""
Jul 27 20:51:35.614672 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.614660    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:51:35.616095 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.615211    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:35.616333 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.616296    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:35.616726 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.616712    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal"]
Jul 27 20:51:35.638850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.638814    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfnmd"
Jul 27 20:51:35.652646 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.652610    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfnmd"
Jul 27 20:51:35.692170 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.692144    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:51:35.697154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-dbus\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.697154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab66c4e-4a49-4752-9050-e2a53b6f9813-host\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.697435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqfx\" (UniqueName: \"kubernetes.io/projected/b1992c52-3136-473a-80ce-bee25c04104a-kube-api-access-mcqfx\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.697435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-cnibin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.697435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3d573c1-2c7d-493e-bb05-f5c607222edd-konnectivity-ca\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.697435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.697435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-sys\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-socket-dir-parent\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-k8s-cni-cncf-io\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-ovn\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-system-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-cni-binary-copy\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.697642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-device-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrgx\" (UniqueName: \"kubernetes.io/projected/43894920-1dc9-4687-ad29-f55409631161-kube-api-access-dlrgx\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-kubernetes\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhg2t\" (UniqueName: \"kubernetes.io/projected/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-kube-api-access-nhg2t\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697799    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-script-lib\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxcfr\" (UniqueName: \"kubernetes.io/projected/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kube-api-access-cxcfr\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpq6\" (UniqueName: \"kubernetes.io/projected/3401dd96-76a0-49f2-8934-ceed9c91835c-kube-api-access-btpq6\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-conf\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.697910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-config\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-os-release\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-hostroot\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.697981    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-bin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-slash\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698139    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-log-socket\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-run\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-etc-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-bin\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-multus-daemon-config\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-iptables-alerter-script\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-os-release\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-host\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-host-slash\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3d573c1-2c7d-493e-bb05-f5c607222edd-agent-certs\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9hsk\" (UniqueName: \"kubernetes.io/projected/10344088-6c8e-4058-bd0b-caf6897c208c-kube-api-access-k9hsk\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysconfig\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698760    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmlc\" (UniqueName: \"kubernetes.io/projected/d3f50019-4b78-4178-b075-f78a0636d68b-kube-api-access-qrmlc\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bab66c4e-4a49-4752-9050-e2a53b6f9813-serviceca\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698822    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-multus-certs\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43894920-1dc9-4687-ad29-f55409631161-hosts-file\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43894920-1dc9-4687-ad29-f55409631161-tmp-dir\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.698948 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-system-cni-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.698948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-netns\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-netns\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-cnibin\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699201    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-kubelet-config\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-lib-modules\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfhq\" (UniqueName: \"kubernetes.io/projected/bab66c4e-4a49-4752-9050-e2a53b6f9813-kube-api-access-6tfhq\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-kubelet\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-kubelet\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-var-lib-kubelet\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-node-log\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-multus\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.699520 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8278d\" (UniqueName: \"kubernetes.io/projected/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-kube-api-access-8278d\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-env-overrides\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1992c52-3136-473a-80ce-bee25c04104a-ovn-node-metrics-cert\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-modprobe-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699478    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-tuned\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-tmp\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-systemd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-systemd\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-var-lib-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-conf-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-etc-kubernetes\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-systemd-units\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-netd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.700231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.699758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.800150 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bab66c4e-4a49-4752-9050-e2a53b6f9813-serviceca\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.800361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800160    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-multus-certs\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.800361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800187    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43894920-1dc9-4687-ad29-f55409631161-hosts-file\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.800361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43894920-1dc9-4687-ad29-f55409631161-tmp-dir\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.800661 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-multus-certs\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.800822 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-system-cni-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.800936 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.801014 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.800976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43894920-1dc9-4687-ad29-f55409631161-tmp-dir\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.801261 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-system-cni-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.801400 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/43894920-1dc9-4687-ad29-f55409631161-hosts-file\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.801469 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.801397    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:35.801537 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:35.801604 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.801575    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:36.301555048 +0000 UTC m=+2.075487467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:35.801604 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-netns\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.801740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.801740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-netns\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.801740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.801899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-cnibin\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.801899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-netns\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.801899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-kubelet-config\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.801899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bab66c4e-4a49-4752-9050-e2a53b6f9813-serviceca\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.801899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-netns\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-kubelet-config\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.801924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-lib-modules\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfhq\" (UniqueName: \"kubernetes.io/projected/bab66c4e-4a49-4752-9050-e2a53b6f9813-kube-api-access-6tfhq\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802065    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-cnibin\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-kubelet\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-lib-modules\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.802154 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-kubelet\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-kubelet\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-var-lib-kubelet\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-node-log\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-kubelet\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-multus\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-var-lib-kubelet\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.802612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8278d\" (UniqueName: \"kubernetes.io/projected/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-kube-api-access-8278d\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802620    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-multus\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-node-log\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-env-overrides\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1992c52-3136-473a-80ce-bee25c04104a-ovn-node-metrics-cert\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.802908 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.802826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-modprobe-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.803220 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-modprobe-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.803281 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803240    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-tuned\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-tmp\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-systemd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-systemd\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803543    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-var-lib-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-conf-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-env-overrides\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-etc-kubernetes\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-systemd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-systemd-units\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803722    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-var-lib-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-systemd-units\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-systemd\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-conf-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-etc-kubernetes\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.808584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-netd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.803940    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.803983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-dbus\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab66c4e-4a49-4752-9050-e2a53b6f9813-host\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.804056    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:36.30403516 +0000 UTC m=+2.077967578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bab66c4e-4a49-4752-9050-e2a53b6f9813-host\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqfx\" (UniqueName: \"kubernetes.io/projected/b1992c52-3136-473a-80ce-bee25c04104a-kube-api-access-mcqfx\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-cnibin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-netd\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3d573c1-2c7d-493e-bb05-f5c607222edd-konnectivity-ca\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-dbus\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-d\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804756    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-cnibin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-sys\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.809602 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-socket-dir-parent\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-k8s-cni-cncf-io\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.804931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-ovn\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-system-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-cni-binary-copy\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-device-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805251    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrgx\" (UniqueName: \"kubernetes.io/projected/43894920-1dc9-4687-ad29-f55409631161-kube-api-access-dlrgx\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3d573c1-2c7d-493e-bb05-f5c607222edd-konnectivity-ca\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-kubernetes\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhg2t\" (UniqueName: \"kubernetes.io/projected/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-kube-api-access-nhg2t\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805377    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-kubernetes\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-script-lib\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxcfr\" (UniqueName: \"kubernetes.io/projected/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kube-api-access-cxcfr\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpq6\" (UniqueName: \"kubernetes.io/projected/3401dd96-76a0-49f2-8934-ceed9c91835c-kube-api-access-btpq6\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-sys\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-conf\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.810684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805561    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-config\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-os-release\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-socket-dir-parent\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-hostroot\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-run-k8s-cni-cncf-io\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-bin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-slash\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.805994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-log-socket\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806044    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-log-socket\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-run-ovn\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-system-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.811581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806478    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-device-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-run\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-etc-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-bin\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-multus-daemon-config\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-iptables-alerter-script\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-os-release\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-host\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-host-slash\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3d573c1-2c7d-493e-bb05-f5c607222edd-agent-certs\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9hsk\" (UniqueName: \"kubernetes.io/projected/10344088-6c8e-4058-bd0b-caf6897c208c-kube-api-access-k9hsk\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysconfig\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmlc\" (UniqueName: \"kubernetes.io/projected/d3f50019-4b78-4178-b075-f78a0636d68b-kube-api-access-qrmlc\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.812485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-script-lib\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-cni-bin\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-host-var-lib-cni-bin\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-cni-binary-copy\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-socket-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-slash\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-tmp\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807515    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-multus-cni-dir\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-registration-dir\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10344088-6c8e-4058-bd0b-caf6897c208c-os-release\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-os-release\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d3f50019-4b78-4178-b075-f78a0636d68b-hostroot\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.806431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-sys-fs\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807869    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-run\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-host-slash\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-host-run-ovn-kubernetes\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.807951    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-host\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.813422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1992c52-3136-473a-80ce-bee25c04104a-etc-openvswitch\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysctl-conf\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-sysconfig\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10344088-6c8e-4058-bd0b-caf6897c208c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808475    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-iptables-alerter-script\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.808685    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d3f50019-4b78-4178-b075-f78a0636d68b-multus-daemon-config\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.808718    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.808733    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:35.808896    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:36.30887267 +0000 UTC m=+2.082805087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1992c52-3136-473a-80ce-bee25c04104a-ovn-node-metrics-cert\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.808953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1992c52-3136-473a-80ce-bee25c04104a-ovnkube-config\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.809817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-etc-tuned\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.811176    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3d573c1-2c7d-493e-bb05-f5c607222edd-agent-certs\") pod \"konnectivity-agent-b82kq\" (UID: \"d3d573c1-2c7d-493e-bb05-f5c607222edd\") " pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.812205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8278d\" (UniqueName: \"kubernetes.io/projected/28f2c7ca-e94e-4082-b5f3-9ffc07b367ab-kube-api-access-8278d\") pod \"iptables-alerter-4qglz\" (UID: \"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab\") " pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:35.814391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.812317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfhq\" (UniqueName: \"kubernetes.io/projected/bab66c4e-4a49-4752-9050-e2a53b6f9813-kube-api-access-6tfhq\") pod \"node-ca-jnmb6\" (UID: \"bab66c4e-4a49-4752-9050-e2a53b6f9813\") " pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.815220 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.815192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqfx\" (UniqueName: \"kubernetes.io/projected/b1992c52-3136-473a-80ce-bee25c04104a-kube-api-access-mcqfx\") pod \"ovnkube-node-xk7hl\" (UID: \"b1992c52-3136-473a-80ce-bee25c04104a\") " pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:35.815909 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.815891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhg2t\" (UniqueName: \"kubernetes.io/projected/6c0c83b0-383e-40ca-adea-ad54cd5bd51a-kube-api-access-nhg2t\") pod \"tuned-t77v8\" (UID: \"6c0c83b0-383e-40ca-adea-ad54cd5bd51a\") " pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.817780 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.817725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrgx\" (UniqueName: \"kubernetes.io/projected/43894920-1dc9-4687-ad29-f55409631161-kube-api-access-dlrgx\") pod \"node-resolver-52wdx\" (UID: \"43894920-1dc9-4687-ad29-f55409631161\") " pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:35.818583 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.818549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxcfr\" (UniqueName: \"kubernetes.io/projected/bef0dacf-58d3-4552-8a62-308e4b8cc3e5-kube-api-access-cxcfr\") pod \"aws-ebs-csi-driver-node-qzqxx\" (UID: \"bef0dacf-58d3-4552-8a62-308e4b8cc3e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.819401 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.819375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpq6\" (UniqueName: \"kubernetes.io/projected/3401dd96-76a0-49f2-8934-ceed9c91835c-kube-api-access-btpq6\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:35.820034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.820010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9hsk\" (UniqueName: \"kubernetes.io/projected/10344088-6c8e-4058-bd0b-caf6897c208c-kube-api-access-k9hsk\") pod \"multus-additional-cni-plugins-dwpt7\" (UID: \"10344088-6c8e-4058-bd0b-caf6897c208c\") " pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:35.820101 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.820082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmlc\" (UniqueName: \"kubernetes.io/projected/d3f50019-4b78-4178-b075-f78a0636d68b-kube-api-access-qrmlc\") pod \"multus-lwpmh\" (UID: \"d3f50019-4b78-4178-b075-f78a0636d68b\") " pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:35.821606 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.821562    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336dbe90a42e70f001ef4a93975ef496.slice/crio-ad7263b810b4db42bac4c693c9d8212095df887d78710a581ec42744cde979af WatchSource:0}: Error finding container ad7263b810b4db42bac4c693c9d8212095df887d78710a581ec42744cde979af: Status 404 returned error can't find the container with id ad7263b810b4db42bac4c693c9d8212095df887d78710a581ec42744cde979af
Jul 27 20:51:35.823730 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.823709    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbe27aeb4e99555881d264eb730f693.slice/crio-f66f58dfe722e59acd7d59e23bc3a03683fca3851f20e0f79dc1f73d1d9ca9b8 WatchSource:0}: Error finding container f66f58dfe722e59acd7d59e23bc3a03683fca3851f20e0f79dc1f73d1d9ca9b8: Status 404 returned error can't find the container with id f66f58dfe722e59acd7d59e23bc3a03683fca3851f20e0f79dc1f73d1d9ca9b8
Jul 27 20:51:35.826431 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.826413    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:51:35.875304 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.875262    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:35.922482 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.922374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:35.928849 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.928820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx"
Jul 27 20:51:35.932503 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.932475    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d573c1_2c7d_493e_bb05_f5c607222edd.slice/crio-6bc96facb46fc225aa60c533e6f91b04ce3419665582687a9ede5383ed4a139b WatchSource:0}: Error finding container 6bc96facb46fc225aa60c533e6f91b04ce3419665582687a9ede5383ed4a139b: Status 404 returned error can't find the container with id 6bc96facb46fc225aa60c533e6f91b04ce3419665582687a9ede5383ed4a139b
Jul 27 20:51:35.937945 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.937922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef0dacf_58d3_4552_8a62_308e4b8cc3e5.slice/crio-23c2fb1e7b32555028229a6414cd4b69aa445f2984275df8aec8721a08e64edf WatchSource:0}: Error finding container 23c2fb1e7b32555028229a6414cd4b69aa445f2984275df8aec8721a08e64edf: Status 404 returned error can't find the container with id 23c2fb1e7b32555028229a6414cd4b69aa445f2984275df8aec8721a08e64edf
Jul 27 20:51:35.949958 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.949931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t77v8"
Jul 27 20:51:35.958309 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.958270    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0c83b0_383e_40ca_adea_ad54cd5bd51a.slice/crio-271d395dcc90dd202c974810065ce55637cea25d0f29f71b88944fff9ba07432 WatchSource:0}: Error finding container 271d395dcc90dd202c974810065ce55637cea25d0f29f71b88944fff9ba07432: Status 404 returned error can't find the container with id 271d395dcc90dd202c974810065ce55637cea25d0f29f71b88944fff9ba07432
Jul 27 20:51:35.962860 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:35.962838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jnmb6"
Jul 27 20:51:35.972707 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:35.972677    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab66c4e_4a49_4752_9050_e2a53b6f9813.slice/crio-95e60bebe6e6d45d95df8a3792e5a203cd8e4798f0763f2f51be4c03f4800b53 WatchSource:0}: Error finding container 95e60bebe6e6d45d95df8a3792e5a203cd8e4798f0763f2f51be4c03f4800b53: Status 404 returned error can't find the container with id 95e60bebe6e6d45d95df8a3792e5a203cd8e4798f0763f2f51be4c03f4800b53
Jul 27 20:51:36.004205 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.004170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:51:36.012615 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:36.012578    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1992c52_3136_473a_80ce_bee25c04104a.slice/crio-6dccc73d6f33ad98ba6c1a1f03ec2dd2685bc3601e91a31a7b38ba1bb3f7cb0c WatchSource:0}: Error finding container 6dccc73d6f33ad98ba6c1a1f03ec2dd2685bc3601e91a31a7b38ba1bb3f7cb0c: Status 404 returned error can't find the container with id 6dccc73d6f33ad98ba6c1a1f03ec2dd2685bc3601e91a31a7b38ba1bb3f7cb0c
Jul 27 20:51:36.025326 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.025300    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52wdx"
Jul 27 20:51:36.033114 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:36.033086    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43894920_1dc9_4687_ad29_f55409631161.slice/crio-11d0c75797c29bd4dcbca83a8afce30ac049d732a6dfd2bc1c0d0e8027c2b0a0 WatchSource:0}: Error finding container 11d0c75797c29bd4dcbca83a8afce30ac049d732a6dfd2bc1c0d0e8027c2b0a0: Status 404 returned error can't find the container with id 11d0c75797c29bd4dcbca83a8afce30ac049d732a6dfd2bc1c0d0e8027c2b0a0
Jul 27 20:51:36.039731 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.039708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwpt7"
Jul 27 20:51:36.047515 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:36.047491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10344088_6c8e_4058_bd0b_caf6897c208c.slice/crio-e2f5258f56e68a6b747489cc840adb645c5599e45d2cc40df9677366f35f6362 WatchSource:0}: Error finding container e2f5258f56e68a6b747489cc840adb645c5599e45d2cc40df9677366f35f6362: Status 404 returned error can't find the container with id e2f5258f56e68a6b747489cc840adb645c5599e45d2cc40df9677366f35f6362
Jul 27 20:51:36.054518 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.054492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lwpmh"
Jul 27 20:51:36.061869 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.061850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4qglz"
Jul 27 20:51:36.063369 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:36.063344    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f50019_4b78_4178_b075_f78a0636d68b.slice/crio-cd58d9495e9ac97e217e3d0704dc1e4cad128610066d14bc76bf526babf61bec WatchSource:0}: Error finding container cd58d9495e9ac97e217e3d0704dc1e4cad128610066d14bc76bf526babf61bec: Status 404 returned error can't find the container with id cd58d9495e9ac97e217e3d0704dc1e4cad128610066d14bc76bf526babf61bec
Jul 27 20:51:36.069774 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:51:36.069750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f2c7ca_e94e_4082_b5f3_9ffc07b367ab.slice/crio-78c33f409196d71c6e3299ae5cf5aeba6c57494dd7a9718bf42577b8ed875d82 WatchSource:0}: Error finding container 78c33f409196d71c6e3299ae5cf5aeba6c57494dd7a9718bf42577b8ed875d82: Status 404 returned error can't find the container with id 78c33f409196d71c6e3299ae5cf5aeba6c57494dd7a9718bf42577b8ed875d82
Jul 27 20:51:36.129636 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.129599    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:36.311695 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.311654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:36.311895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.311775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:36.311895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.311807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:36.311895 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311878    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311913    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311952    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:37.311931101 +0000 UTC m=+3.085863523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311957    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:37.311962701 +0000 UTC m=+3.085895102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311977    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.311992    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:36.312088 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.312050    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:37.312032818 +0000 UTC m=+3.085965236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:36.429897 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.429842    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:36.655436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.655272    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:36 +0000 UTC" deadline="2028-05-07 07:39:52.165177694 +0000 UTC"
Jul 27 20:51:36.655436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.655313    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15586h48m15.509870013s"
Jul 27 20:51:36.799380 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.798752    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:36.799380 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:36.798973    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:36.825148 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.825071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b82kq" event={"ID":"d3d573c1-2c7d-493e-bb05-f5c607222edd","Type":"ContainerStarted","Data":"6bc96facb46fc225aa60c533e6f91b04ce3419665582687a9ede5383ed4a139b"}
Jul 27 20:51:36.848425 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.848382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"336dbe90a42e70f001ef4a93975ef496","Type":"ContainerStarted","Data":"ad7263b810b4db42bac4c693c9d8212095df887d78710a581ec42744cde979af"}
Jul 27 20:51:36.868322 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.868237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lwpmh" event={"ID":"d3f50019-4b78-4178-b075-f78a0636d68b","Type":"ContainerStarted","Data":"cd58d9495e9ac97e217e3d0704dc1e4cad128610066d14bc76bf526babf61bec"}
Jul 27 20:51:36.887893 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.887843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52wdx" event={"ID":"43894920-1dc9-4687-ad29-f55409631161","Type":"ContainerStarted","Data":"11d0c75797c29bd4dcbca83a8afce30ac049d732a6dfd2bc1c0d0e8027c2b0a0"}
Jul 27 20:51:36.902439 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.902313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"6dccc73d6f33ad98ba6c1a1f03ec2dd2685bc3601e91a31a7b38ba1bb3f7cb0c"}
Jul 27 20:51:36.950166 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.949931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t77v8" event={"ID":"6c0c83b0-383e-40ca-adea-ad54cd5bd51a","Type":"ContainerStarted","Data":"271d395dcc90dd202c974810065ce55637cea25d0f29f71b88944fff9ba07432"}
Jul 27 20:51:36.963395 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.961188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" event={"ID":"9cbe27aeb4e99555881d264eb730f693","Type":"ContainerStarted","Data":"f66f58dfe722e59acd7d59e23bc3a03683fca3851f20e0f79dc1f73d1d9ca9b8"}
Jul 27 20:51:36.989676 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:36.989630    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qglz" event={"ID":"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab","Type":"ContainerStarted","Data":"78c33f409196d71c6e3299ae5cf5aeba6c57494dd7a9718bf42577b8ed875d82"}
Jul 27 20:51:37.019671 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.019592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerStarted","Data":"e2f5258f56e68a6b747489cc840adb645c5599e45d2cc40df9677366f35f6362"}
Jul 27 20:51:37.027585 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.027411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnmb6" event={"ID":"bab66c4e-4a49-4752-9050-e2a53b6f9813","Type":"ContainerStarted","Data":"95e60bebe6e6d45d95df8a3792e5a203cd8e4798f0763f2f51be4c03f4800b53"}
Jul 27 20:51:37.030172 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.030082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx" event={"ID":"bef0dacf-58d3-4552-8a62-308e4b8cc3e5","Type":"ContainerStarted","Data":"23c2fb1e7b32555028229a6414cd4b69aa445f2984275df8aec8721a08e64edf"}
Jul 27 20:51:37.321566 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.321509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:37.321769 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.321616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:37.321769 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.321647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:37.321890 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.321810    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:37.321890 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.321830    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:37.321890 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.321844    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:37.322047 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.321910    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.321889503 +0000 UTC m=+5.095821907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:37.322515 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.322361    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:37.322515 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.322417    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.32240087 +0000 UTC m=+5.096333272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:37.322515 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.322480    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:37.322515 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.322513    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.322502484 +0000 UTC m=+5.096434892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:37.431788 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.431754    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:37.656144 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.656019    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:36 +0000 UTC" deadline="2028-03-30 15:17:51.218112764 +0000 UTC"
Jul 27 20:51:37.656144 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.656088    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14682h26m13.562029186s"
Jul 27 20:51:37.799237 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.799180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:37.799459 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.799350    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:37.799850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:37.799800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:37.799938 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:37.799905    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:38.806318 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:38.806269    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:38.806843 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:38.806448    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:39.344685 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:39.344645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:39.344877 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:39.344696    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:39.344877 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:39.344742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:39.345019 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.344904    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:39.345019 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.344973    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:43.344952906 +0000 UTC m=+9.118885312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:39.345463 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345407    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:39.345558 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:43.345453426 +0000 UTC m=+9.119385827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:39.345628 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345558    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:39.345628 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345573    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:39.345628 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345585    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:39.345628 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.345622    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:43.345611133 +0000 UTC m=+9.119543537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:39.800332 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:39.799559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:39.800332 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.799706    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:39.800332 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:39.800111    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:39.800332 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:39.800219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:40.804618 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:40.804583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:40.805127 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:40.804756    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:41.799881 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:41.799142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:41.799881 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:41.799284    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:41.799881 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:41.799736    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:41.799881 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:41.799829    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:42.799573 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:42.799534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:42.800046 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:42.799672    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:43.381842 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:43.381800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:43.382066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:43.381855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:43.382066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:43.381899    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:43.382066 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.381986    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:43.382066 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382018    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382079    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:51.382053638 +0000 UTC m=+17.155986048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382102    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:51:51.382091131 +0000 UTC m=+17.156023532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382147    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382161    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382172    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:43.382315 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.382201    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:51.382189201 +0000 UTC m=+17.156121600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:43.799415 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:43.799375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:43.799634 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.799513    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:43.799938 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:43.799826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:43.799938 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:43.799897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:44.802961 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:44.802924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:44.803464 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:44.803066    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:45.798903 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:45.798866    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:45.799090 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:45.798994    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:45.799531 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:45.799515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:45.799622 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:45.799601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:46.799374 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:46.799340    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:46.799808 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:46.799480    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:47.799348 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:47.799310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:47.799538 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:47.799311    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:47.799538 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:47.799434    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:47.799925 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:47.799549    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:48.811901 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:48.811826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:48.812285 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:48.811938    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:49.798832 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:49.798796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:49.799020 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:49.798799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:49.799020 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:49.798957    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:49.799020 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:49.798990    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:50.799509 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:50.799465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:50.799922 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:50.799618    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:51.450108 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:51.450061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:51.450108 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:51.450110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:51.450139    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450252    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450265    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450285    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450303    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450315    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450334    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:52:07.450312591 +0000 UTC m=+33.224244989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450356    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:52:07.450345919 +0000 UTC m=+33.224278324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:51.450403 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.450371    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:07.450363186 +0000 UTC m=+33.224295588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:51.798800 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:51.798761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:51.798800 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:51.798797    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:51.799095 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.798892    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:51.799095 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:51.799011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:52.799573 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:52.799478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:52.800025 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:52.799627    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:53.799113 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:53.799072    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:53.799323 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:53.799073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:53.799323 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:53.799194    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:53.799420 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:53.799316    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:54.799722 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:54.799690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:54.800187 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:54.799808    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:55.799703 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:55.799665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:55.799910 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:55.799784    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:55.799910 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:55.799854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:55.800345 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:55.799925    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:56.799884 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:56.799611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:56.800034 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:56.799942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:57.081890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.081789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lwpmh" event={"ID":"d3f50019-4b78-4178-b075-f78a0636d68b","Type":"ContainerStarted","Data":"bababb7a742307954b8f0071769cc4c338f5a458ec829694e51f79a15c983473"}
Jul 27 20:51:57.084550 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.084518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t77v8" event={"ID":"6c0c83b0-383e-40ca-adea-ad54cd5bd51a","Type":"ContainerStarted","Data":"a794ebb3677ffa8d0a77bf1c517a651d264c09e1f3ea42677d91f4f2ecf6e03d"}
Jul 27 20:51:57.086383 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.086357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" event={"ID":"9cbe27aeb4e99555881d264eb730f693","Type":"ContainerStarted","Data":"a7cdc57c39cd5099959a9ec42fcf55b81d730154b4d4ee76096ec8239d413ae0"}
Jul 27 20:51:57.101949 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.101900    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lwpmh" podStartSLOduration=1.361202595 podStartE2EDuration="22.101884628s" totalImagesPullingTime="20.740682033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:57.101432209 +0000 UTC m=+22.875364632" watchObservedRunningTime="2026-07-27 20:51:57.101884628 +0000 UTC m=+22.875817048"
Jul 27 20:51:57.121556 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.121500    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t77v8" podStartSLOduration=1.5600677649999999 podStartE2EDuration="22.121481201s" totalImagesPullingTime="20.561413436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:57.120895834 +0000 UTC m=+22.894828261" watchObservedRunningTime="2026-07-27 20:51:57.121481201 +0000 UTC m=+22.895413629"
Jul 27 20:51:57.138439 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.138372    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-89.ec2.internal" podStartSLOduration=21.138348591 podStartE2EDuration="21.138348591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:57.137092047 +0000 UTC m=+22.911024467" watchObservedRunningTime="2026-07-27 20:51:57.138348591 +0000 UTC m=+22.912281017"
Jul 27 20:51:57.799173 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.798873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:57.799365 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:57.798873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:57.799365 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:57.799288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:57.799365 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:57.799346    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:51:58.089672 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.089626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52wdx" event={"ID":"43894920-1dc9-4687-ad29-f55409631161","Type":"ContainerStarted","Data":"31ec25b582225b249350b5ef98e46c9e35a2ca2651aee009ab50f02b398a73cd"}
Jul 27 20:51:58.092586 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.092521    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:51:58.092906 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.092882    2580 generic.go:356] "Generic (PLEG): container finished" podID="b1992c52-3136-473a-80ce-bee25c04104a" containerID="a9b4bbb72956c79218f976db697f44abda386a9846fabd0f7c68706c1df5133a" exitCode=1
Jul 27 20:51:58.093015 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.092947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"f1f72ee583fe500ca1ae978682bf9735ab7e6b6ea282fc4e4ff3b98ee4f68022"}
Jul 27 20:51:58.093094 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.093028    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"d48b680d055aa098e3e71e8593b00f94a8db10c57c99fba2e1736b43c46bc8e0"}
Jul 27 20:51:58.093094 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.093079    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"13236970a69b9c040ef24273e842237c5a092e0d03928ee588d2a0b9af281035"}
Jul 27 20:51:58.093169 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.093125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"849f7efa1e53741259dc26794e0c340182d49f3984b776d96e7262a8dc94f6d0"}
Jul 27 20:51:58.093225 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.093198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerDied","Data":"a9b4bbb72956c79218f976db697f44abda386a9846fabd0f7c68706c1df5133a"}
Jul 27 20:51:58.093279 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.093272    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"3f7d5a0d22c733cda38c78fd89ee2ffc3e1be68ca6c54c0ecc55deabaf148fe2"}
Jul 27 20:51:58.094529 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.094508    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="14140d7612fdbc1e92c07fb0e0834faeb71c9cc12844cda7f50d7e235af280f1" exitCode=0
Jul 27 20:51:58.094612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.094574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"14140d7612fdbc1e92c07fb0e0834faeb71c9cc12844cda7f50d7e235af280f1"}
Jul 27 20:51:58.096197 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.096036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jnmb6" event={"ID":"bab66c4e-4a49-4752-9050-e2a53b6f9813","Type":"ContainerStarted","Data":"b16f1d22a2bcc2af76b1368a119b368f26105c988b88bb162e3575d851f47033"}
Jul 27 20:51:58.097618 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.097592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx" event={"ID":"bef0dacf-58d3-4552-8a62-308e4b8cc3e5","Type":"ContainerStarted","Data":"265a9c38c60aaf31e3707c5d7d1360e1550bfd5658fd1b06f63b80f97fa79a32"}
Jul 27 20:51:58.099233 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.099191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b82kq" event={"ID":"d3d573c1-2c7d-493e-bb05-f5c607222edd","Type":"ContainerStarted","Data":"a82655f8ba457c6bdf7887060c8ecc2230893037354558b7a3aea71ef74accd1"}
Jul 27 20:51:58.101160 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.101126    2580 generic.go:356] "Generic (PLEG): container finished" podID="336dbe90a42e70f001ef4a93975ef496" containerID="823908260c6894ad6c2338312c7278866eec7635b40c13d0c93424935d39ab3f" exitCode=0
Jul 27 20:51:58.101365 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.101322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"336dbe90a42e70f001ef4a93975ef496","Type":"ContainerDied","Data":"823908260c6894ad6c2338312c7278866eec7635b40c13d0c93424935d39ab3f"}
Jul 27 20:51:58.113657 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.113601    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-52wdx" podStartSLOduration=2.628448805 podStartE2EDuration="23.113586083s" totalImagesPullingTime="20.485137278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:58.113186757 +0000 UTC m=+23.887119176" watchObservedRunningTime="2026-07-27 20:51:58.113586083 +0000 UTC m=+23.887518503"
Jul 27 20:51:58.154777 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.154722    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b82kq" podStartSLOduration=2.606195418 podStartE2EDuration="23.154705107s" totalImagesPullingTime="20.548509689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:58.154434533 +0000 UTC m=+23.928366948" watchObservedRunningTime="2026-07-27 20:51:58.154705107 +0000 UTC m=+23.928637551"
Jul 27 20:51:58.171490 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.171433    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jnmb6" podStartSLOduration=2.6624585830000003 podStartE2EDuration="23.171412526s" totalImagesPullingTime="20.508953943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:58.170628566 +0000 UTC m=+23.944560987" watchObservedRunningTime="2026-07-27 20:51:58.171412526 +0000 UTC m=+23.945344947"
Jul 27 20:51:58.264529 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.264492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:58.270449 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.270424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:58.270982 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.270964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:58.338977 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.338906    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:58.611436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.611408    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:51:58.672459 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.672345    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:51:58.611431174Z","UUID":"f3f1f45c-489d-4d3f-bbea-37578877423f","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:51:58.673875 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.673854    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:51:58.673981 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.673895    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:51:58.799537 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:58.799409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:51:58.799699 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:58.799554    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:51:59.105705 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.105649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4qglz" event={"ID":"28f2c7ca-e94e-4082-b5f3-9ffc07b367ab","Type":"ContainerStarted","Data":"131b78c28bbf1b0b2196db4f63bc8eeaddd277f800a1131239cdb6bcce054541"}
Jul 27 20:51:59.107767 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.107740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx" event={"ID":"bef0dacf-58d3-4552-8a62-308e4b8cc3e5","Type":"ContainerStarted","Data":"bd04f943cad81f17adc921ec76afb3940448c93296e06c746dac21f27208169e"}
Jul 27 20:51:59.109744 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.109710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" event={"ID":"336dbe90a42e70f001ef4a93975ef496","Type":"ContainerStarted","Data":"87b5343626b0e0309200e27bc41a5b3cf462bdaceee9a8e0434319a658a9c95c"}
Jul 27 20:51:59.109955 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.109932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:59.110394 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.110378    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b82kq"
Jul 27 20:51:59.136066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.136015    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4qglz" podStartSLOduration=3.687405905 podStartE2EDuration="24.135999553s" totalImagesPullingTime="20.448593648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:59.133562278 +0000 UTC m=+24.907494707" watchObservedRunningTime="2026-07-27 20:51:59.135999553 +0000 UTC m=+24.909931973"
Jul 27 20:51:59.178599 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.178534    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-89.ec2.internal" podStartSLOduration=23.178510137 podStartE2EDuration="23.178510137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:59.177774243 +0000 UTC m=+24.951706666" watchObservedRunningTime="2026-07-27 20:51:59.178510137 +0000 UTC m=+24.952442562"
Jul 27 20:51:59.220879 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.220843    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.293016 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.292972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.372752 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.372626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.798870 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.798836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:51:59.799058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:51:59.798836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:51:59.799058 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:59.798978    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:51:59.799158 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:51:59.799056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:00.118039 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:00.117996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx" event={"ID":"bef0dacf-58d3-4552-8a62-308e4b8cc3e5","Type":"ContainerStarted","Data":"0b3e728fb74de5f76bcc28d8dd4bded5b2af9419d23b6b971e423a612905ec8d"}
Jul 27 20:52:00.798962 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:00.798923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:00.799150 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:00.799076    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:01.123257 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.123169    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:52:01.123673 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.123651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"484e692d3ff1da4d86eaf31b96070dcc4d1c7dc85d06ed9e70d200af17623f13"}
Jul 27 20:52:01.144015 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.143963    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qzqxx" podStartSLOduration=3.25698678 podStartE2EDuration="26.143944695s" totalImagesPullingTime="22.886957915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:01.143367655 +0000 UTC m=+26.917300076" watchObservedRunningTime="2026-07-27 20:52:01.143944695 +0000 UTC m=+26.917877116"
Jul 27 20:52:01.263890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.263851    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:01.363700 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.363668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:01.799482 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.799444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:01.799680 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:01.799444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:01.799680 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:01.799564    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:01.799806 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:01.799679    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:02.799320 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:02.799071    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:02.799905 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:02.799371    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:03.132273 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.132177    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:52:03.132580 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.132556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"97230ca46b722f1b07a37d85971da845bd14d29349ef5c3d49a745251c8a2f2d"}
Jul 27 20:52:03.132901 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.132877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:03.133083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.133065    2580 scope.go:117] "RemoveContainer" containerID="a9b4bbb72956c79218f976db697f44abda386a9846fabd0f7c68706c1df5133a"
Jul 27 20:52:03.134411 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.134391    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="54691a0a3274e1089bfd5a72fc3bc5da31f64556eddf7497705ccd36756a6b65" exitCode=0
Jul 27 20:52:03.134506 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.134433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"54691a0a3274e1089bfd5a72fc3bc5da31f64556eddf7497705ccd36756a6b65"}
Jul 27 20:52:03.151320 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.151295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:03.803091 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.803060    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:03.803579 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:03.803183    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:03.803635 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:03.803582    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:03.803683 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:03.803666    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:04.052916 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.052884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:04.144928 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.144838    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:52:04.145809 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.145692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" event={"ID":"b1992c52-3136-473a-80ce-bee25c04104a","Type":"ContainerStarted","Data":"6555b9b5db834a14f3e98caba847f9d9e696fce7938beec427b669baedaf7461"}
Jul 27 20:52:04.151711 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.151513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:04.151711 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.151600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:04.154588 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.154553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:04.175156 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.175078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:04.194947 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.194894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl" podStartSLOduration=8.003544914 podStartE2EDuration="29.194877349s" totalImagesPullingTime="21.191332435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:04.19404515 +0000 UTC m=+29.967977571" watchObservedRunningTime="2026-07-27 20:52:04.194877349 +0000 UTC m=+29.968809769"
Jul 27 20:52:04.347913 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.347882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m582w"]
Jul 27 20:52:04.348073 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.348006    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:04.348128 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:04.348100    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:04.351093 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.351069    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pxcw2"]
Jul 27 20:52:04.351255 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.351185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:04.351326 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:04.351304    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:04.351684 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.351665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fd7cv"]
Jul 27 20:52:04.351762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:04.351750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:04.351874 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:04.351836    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:05.150053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:05.150017    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="ac32a0a5dae6442ec162cfd29306bfb18f94e04186538beb2131267d7edc5fdb" exitCode=0
Jul 27 20:52:05.150645 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:05.150107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"ac32a0a5dae6442ec162cfd29306bfb18f94e04186538beb2131267d7edc5fdb"}
Jul 27 20:52:05.799504 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:05.799462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:05.799504 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:05.799476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:05.799753 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:05.799486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:05.799753 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:05.799621    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:05.799753 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:05.799682    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:05.799882 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:05.799746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:07.157278 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.156990    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="303ac57a35c258bc4f78422c52c0ff6fde9940f4ac44471eb22f126dfd3bbaaa" exitCode=0
Jul 27 20:52:07.157649 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.157061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"303ac57a35c258bc4f78422c52c0ff6fde9940f4ac44471eb22f126dfd3bbaaa"}
Jul 27 20:52:07.473053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.473020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.473058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.473093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473166    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473176    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473179    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473199    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:52:07.473248 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473209    2580 projected.go:196] Error preparing data for projected volume kube-api-access-9hdj2 for pod openshift-network-diagnostics/network-check-target-pxcw2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:52:07.473482 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473253    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs podName:3401dd96-76a0-49f2-8934-ceed9c91835c nodeName:}" failed. No retries permitted until 2026-07-27 20:52:39.473233933 +0000 UTC m=+65.247166331 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs") pod "network-metrics-daemon-fd7cv" (UID: "3401dd96-76a0-49f2-8934-ceed9c91835c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:52:07.473482 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473267    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret podName:bd91b7f1-6d7e-4cfd-a400-373508ac1c0c nodeName:}" failed. No retries permitted until 2026-07-27 20:52:39.473261088 +0000 UTC m=+65.247193485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret") pod "global-pull-secret-syncer-m582w" (UID: "bd91b7f1-6d7e-4cfd-a400-373508ac1c0c") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:52:07.473482 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.473278    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2 podName:85d66181-00fe-43e8-abca-193045d85495 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:39.473273044 +0000 UTC m=+65.247205441 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hdj2" (UniqueName: "kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2") pod "network-check-target-pxcw2" (UID: "85d66181-00fe-43e8-abca-193045d85495") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:52:07.799530 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.799444    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:07.799530 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.799492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:07.799530 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:07.799492    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:07.799802 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.799570    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:07.799802 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.799691    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:07.799802 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:07.799786    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:09.799545 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:09.799501    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:09.800120 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:09.799508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:09.800120 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:09.799613    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxcw2" podUID="85d66181-00fe-43e8-abca-193045d85495"
Jul 27 20:52:09.800120 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:09.799517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:09.800120 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:09.799730    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fd7cv" podUID="3401dd96-76a0-49f2-8934-ceed9c91835c"
Jul 27 20:52:09.800120 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:09.799774    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m582w" podUID="bd91b7f1-6d7e-4cfd-a400-373508ac1c0c"
Jul 27 20:52:10.399490 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:10.399452    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.484040 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:10.484001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.095445 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.095417    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-89.ec2.internal" event="NodeReady"
Jul 27 20:52:11.095968 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.095567    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:52:11.151318 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.151184    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5dzsn"]
Jul 27 20:52:11.152929 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.152903    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n62fk"]
Jul 27 20:52:11.153054 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.153026    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.154607 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.154587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.155587 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.155545    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:52:11.155993 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.155978    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:52:11.156083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.156003    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9tww\""
Jul 27 20:52:11.158600 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.158441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lslmx\""
Jul 27 20:52:11.158600 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.158457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:52:11.158748 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.158596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:52:11.158748 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.158701    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:52:11.169782 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.169751    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerStarted","Data":"59a3594dbc36984909c08ec3a60a68aba6d0abe9fb4c0f269a0b0b426c7fca79"}
Jul 27 20:52:11.173300 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.173270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5dzsn"]
Jul 27 20:52:11.173839 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.173817    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n62fk"]
Jul 27 20:52:11.303973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.303930    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ckm\" (UniqueName: \"kubernetes.io/projected/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-kube-api-access-w5ckm\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.303973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.303973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b629b7e-680e-4019-a64e-0dc46331d1b9-tmp-dir\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.304267 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.304043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-cert\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.304267 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.304165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpb8\" (UniqueName: \"kubernetes.io/projected/8b629b7e-680e-4019-a64e-0dc46331d1b9-kube-api-access-4tpb8\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.304267 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.304249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.304403 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.304303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b629b7e-680e-4019-a64e-0dc46331d1b9-config-volume\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.324643 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.324557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.385700 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.385659    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.404639 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-cert\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.404639 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpb8\" (UniqueName: \"kubernetes.io/projected/8b629b7e-680e-4019-a64e-0dc46331d1b9-kube-api-access-4tpb8\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.404847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.404847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b629b7e-680e-4019-a64e-0dc46331d1b9-config-volume\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.404847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ckm\" (UniqueName: \"kubernetes.io/projected/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-kube-api-access-w5ckm\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.404847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.404797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b629b7e-680e-4019-a64e-0dc46331d1b9-tmp-dir\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.405032 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:11.404859    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:11.405032 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:11.404936    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:11.904914713 +0000 UTC m=+37.678847135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:11.405157 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.405078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b629b7e-680e-4019-a64e-0dc46331d1b9-tmp-dir\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.405430 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.405407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8b629b7e-680e-4019-a64e-0dc46331d1b9-config-volume\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.409152 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.409132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-cert\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.417432 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.417403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ckm\" (UniqueName: \"kubernetes.io/projected/f4618ee4-fea8-4ac0-865f-8bf5c5c72c68-kube-api-access-w5ckm\") pod \"ingress-canary-n62fk\" (UID: \"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68\") " pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.418273 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.418246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpb8\" (UniqueName: \"kubernetes.io/projected/8b629b7e-680e-4019-a64e-0dc46331d1b9-kube-api-access-4tpb8\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.469182 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.469141    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n62fk"
Jul 27 20:52:11.612432 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.612150    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n62fk"]
Jul 27 20:52:11.615955 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:11.615927    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4618ee4_fea8_4ac0_865f_8bf5c5c72c68.slice/crio-feb74469bb33d0a943eab892d3ae80c6b53fe8b8e26ea09a41effe60451c95bb WatchSource:0}: Error finding container feb74469bb33d0a943eab892d3ae80c6b53fe8b8e26ea09a41effe60451c95bb: Status 404 returned error can't find the container with id feb74469bb33d0a943eab892d3ae80c6b53fe8b8e26ea09a41effe60451c95bb
Jul 27 20:52:11.799612 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.799551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:11.799762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.799551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:11.799762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.799551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:11.802646 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802618    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:52:11.802790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:52:11.802790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802618    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:52:11.802790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802708    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvgk8\""
Jul 27 20:52:11.802790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802739    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:52:11.802790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.802754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qtl47\""
Jul 27 20:52:11.907804 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:11.907716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:11.907942 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:11.907837    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:11.907942 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:11.907892    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:12.907877327 +0000 UTC m=+38.681809726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:12.176086 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:12.176000    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="59a3594dbc36984909c08ec3a60a68aba6d0abe9fb4c0f269a0b0b426c7fca79" exitCode=0
Jul 27 20:52:12.176788 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:12.176086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"59a3594dbc36984909c08ec3a60a68aba6d0abe9fb4c0f269a0b0b426c7fca79"}
Jul 27 20:52:12.177259 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:12.177238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n62fk" event={"ID":"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68","Type":"ContainerStarted","Data":"feb74469bb33d0a943eab892d3ae80c6b53fe8b8e26ea09a41effe60451c95bb"}
Jul 27 20:52:12.917319 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:12.917281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:12.917501 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:12.917430    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:12.917501 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:12.917495    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:14.917478356 +0000 UTC m=+40.691410755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:13.183304 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:13.183226    2580 generic.go:356] "Generic (PLEG): container finished" podID="10344088-6c8e-4058-bd0b-caf6897c208c" containerID="8eeaf48c1835879f675795e6af34711e5e71efd4cc8e81c3ec8e5f3183b0ba06" exitCode=0
Jul 27 20:52:13.183739 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:13.183306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerDied","Data":"8eeaf48c1835879f675795e6af34711e5e71efd4cc8e81c3ec8e5f3183b0ba06"}
Jul 27 20:52:14.187887 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.187632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n62fk" event={"ID":"f4618ee4-fea8-4ac0-865f-8bf5c5c72c68","Type":"ContainerStarted","Data":"2972c23b50419106fd85c74e717d9da9839f9bdcd41e51fa1ffea7dce097752e"}
Jul 27 20:52:14.191389 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.191331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" event={"ID":"10344088-6c8e-4058-bd0b-caf6897c208c","Type":"ContainerStarted","Data":"8d4c071c47faf9f6719cbf66ceb65f75fbd9209a76766ecc8bac67ac8658b372"}
Jul 27 20:52:14.206780 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.206731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n62fk" podStartSLOduration=0.773546815 podStartE2EDuration="3.206716581s" totalImagesPullingTime="2.433169766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:14.205298564 +0000 UTC m=+39.979230984" watchObservedRunningTime="2026-07-27 20:52:14.206716581 +0000 UTC m=+39.980649021"
Jul 27 20:52:14.236886 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.236833    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dwpt7" podStartSLOduration=8.528883955 podStartE2EDuration="39.236815484s" totalImagesPullingTime="30.707931529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:14.23623257 +0000 UTC m=+40.010164988" watchObservedRunningTime="2026-07-27 20:52:14.236815484 +0000 UTC m=+40.010747904"
Jul 27 20:52:14.275077 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.275039    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"]
Jul 27 20:52:14.280805 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.280769    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7c6b4bf9c-jn462"]
Jul 27 20:52:14.280967 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.280892    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"
Jul 27 20:52:14.283368 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.283347    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jds98\""
Jul 27 20:52:14.286885 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.286863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.289413 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.289396    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 20:52:14.289653 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.289636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 20:52:14.289834 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.289814    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.289834 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.289828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vf267\""
Jul 27 20:52:14.289917 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.289852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 20:52:14.290103 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.290089    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 20:52:14.290419 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.290402    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.293442 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.293421    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"]
Jul 27 20:52:14.328791 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.328061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c6b4bf9c-jn462"]
Jul 27 20:52:14.382785 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.382754    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"]
Jul 27 20:52:14.383604 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.383566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.396629 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.396594    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"]
Jul 27 20:52:14.396749 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.396712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.399486 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.399464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.399596 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.399432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 27 20:52:14.400183 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.400155    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bj5hc\""
Jul 27 20:52:14.400332 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.400318    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 27 20:52:14.400559 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.400540    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.406758 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.406736    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"]
Jul 27 20:52:14.406879 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.406861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.409757 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.409739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g7qcs\""
Jul 27 20:52:14.411911 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.411891    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 27 20:52:14.412013 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.411995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.412083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.412071    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.416196 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.416176    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"]
Jul 27 20:52:14.429621 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.429740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-stats-auth\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.429740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.429740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-default-certificate\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.429860 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptx4\" (UniqueName: \"kubernetes.io/projected/7c2bdc7d-2972-4eb0-a771-1858fa4d313b-kube-api-access-fptx4\") pod \"network-check-source-7d88cb55d8-thw6z\" (UID: \"7c2bdc7d-2972-4eb0-a771-1858fa4d313b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"
Jul 27 20:52:14.429860 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.429843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc45t\" (UniqueName: \"kubernetes.io/projected/72c0faa1-cedb-4a19-838f-6e37a26255e3-kube-api-access-hc45t\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.477125 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.477084    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"]
Jul 27 20:52:14.494744 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.494719    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"]
Jul 27 20:52:14.494880 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.494811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"
Jul 27 20:52:14.499601 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.499575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.500252 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.500230    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.500369 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.500239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8zqpk\""
Jul 27 20:52:14.501788 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.501767    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.531172 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.531350 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-stats-auth\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.531350 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.531350 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.531291    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:14.531350 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.531350 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.531338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:15.031323065 +0000 UTC m=+40.805255463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.531385    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:15.031370536 +0000 UTC m=+40.805302941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-default-certificate\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531432    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68d27cc1-738e-4060-a482-70dac6ca1c6a-config\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvxf\" (UniqueName: \"kubernetes.io/projected/68d27cc1-738e-4060-a482-70dac6ca1c6a-kube-api-access-5mvxf\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptx4\" (UniqueName: \"kubernetes.io/projected/7c2bdc7d-2972-4eb0-a771-1858fa4d313b-kube-api-access-fptx4\") pod \"network-check-source-7d88cb55d8-thw6z\" (UID: \"7c2bdc7d-2972-4eb0-a771-1858fa4d313b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc45t\" (UniqueName: \"kubernetes.io/projected/72c0faa1-cedb-4a19-838f-6e37a26255e3-kube-api-access-hc45t\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d27cc1-738e-4060-a482-70dac6ca1c6a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.531595 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.531587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hm4\" (UniqueName: \"kubernetes.io/projected/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-kube-api-access-d4hm4\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.535410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.535391    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-stats-auth\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.535410 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.535400    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-default-certificate\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.544575 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.544555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc45t\" (UniqueName: \"kubernetes.io/projected/72c0faa1-cedb-4a19-838f-6e37a26255e3-kube-api-access-hc45t\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:14.544938 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.544920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptx4\" (UniqueName: \"kubernetes.io/projected/7c2bdc7d-2972-4eb0-a771-1858fa4d313b-kube-api-access-fptx4\") pod \"network-check-source-7d88cb55d8-thw6z\" (UID: \"7c2bdc7d-2972-4eb0-a771-1858fa4d313b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"
Jul 27 20:52:14.585464 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.585377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.590911 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.590894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"
Jul 27 20:52:14.632539 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.632656 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68d27cc1-738e-4060-a482-70dac6ca1c6a-config\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.632656 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvxf\" (UniqueName: \"kubernetes.io/projected/68d27cc1-738e-4060-a482-70dac6ca1c6a-kube-api-access-5mvxf\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.632656 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmskb\" (UniqueName: \"kubernetes.io/projected/e2c93b3d-1e00-489a-9682-f72c95ac215c-kube-api-access-lmskb\") pod \"volume-data-source-validator-549c6d54cf-98m5s\" (UID: \"e2c93b3d-1e00-489a-9682-f72c95ac215c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"
Jul 27 20:52:14.632783 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.632673    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 20:52:14.632783 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d27cc1-738e-4060-a482-70dac6ca1c6a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.632783 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.632741    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls podName:e5f267ae-85e1-430e-84ff-c99b0c4b5d0a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:15.132721292 +0000 UTC m=+40.906653690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls") pod "cluster-samples-operator-6748489656-9g5vs" (UID: "e5f267ae-85e1-430e-84ff-c99b0c4b5d0a") : secret "samples-operator-tls" not found
Jul 27 20:52:14.632892 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.632784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hm4\" (UniqueName: \"kubernetes.io/projected/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-kube-api-access-d4hm4\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.633256 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.633241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68d27cc1-738e-4060-a482-70dac6ca1c6a-config\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.635006 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.634986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68d27cc1-738e-4060-a482-70dac6ca1c6a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.642796 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.642771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvxf\" (UniqueName: \"kubernetes.io/projected/68d27cc1-738e-4060-a482-70dac6ca1c6a-kube-api-access-5mvxf\") pod \"service-ca-operator-5d6c75f84f-vg56m\" (UID: \"68d27cc1-738e-4060-a482-70dac6ca1c6a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.643563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.643546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hm4\" (UniqueName: \"kubernetes.io/projected/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-kube-api-access-d4hm4\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:14.689317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.689273    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.706932 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.706900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"
Jul 27 20:52:14.733313 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.733285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmskb\" (UniqueName: \"kubernetes.io/projected/e2c93b3d-1e00-489a-9682-f72c95ac215c-kube-api-access-lmskb\") pod \"volume-data-source-validator-549c6d54cf-98m5s\" (UID: \"e2c93b3d-1e00-489a-9682-f72c95ac215c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"
Jul 27 20:52:14.737731 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.737701    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z"]
Jul 27 20:52:14.742131 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:14.742104    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2bdc7d_2972_4eb0_a771_1858fa4d313b.slice/crio-7ecc21b22db28ef13808d447ef9f8520fdfdcaa8efafa2955f86ee2c2b042053 WatchSource:0}: Error finding container 7ecc21b22db28ef13808d447ef9f8520fdfdcaa8efafa2955f86ee2c2b042053: Status 404 returned error can't find the container with id 7ecc21b22db28ef13808d447ef9f8520fdfdcaa8efafa2955f86ee2c2b042053
Jul 27 20:52:14.742259 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.742234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmskb\" (UniqueName: \"kubernetes.io/projected/e2c93b3d-1e00-489a-9682-f72c95ac215c-kube-api-access-lmskb\") pod \"volume-data-source-validator-549c6d54cf-98m5s\" (UID: \"e2c93b3d-1e00-489a-9682-f72c95ac215c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"
Jul 27 20:52:14.804779 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.804746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"
Jul 27 20:52:14.860558 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.860530    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m"]
Jul 27 20:52:14.864281 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:14.864248    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d27cc1_738e_4060_a482_70dac6ca1c6a.slice/crio-048738f228768e988ef5337b5b0b6e05c9eff6c77d8e7d84a951a45c0b40ec98 WatchSource:0}: Error finding container 048738f228768e988ef5337b5b0b6e05c9eff6c77d8e7d84a951a45c0b40ec98: Status 404 returned error can't find the container with id 048738f228768e988ef5337b5b0b6e05c9eff6c77d8e7d84a951a45c0b40ec98
Jul 27 20:52:14.934780 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.934752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:14.934908 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.934865    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:14.934960 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:14.934919    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:18.934904583 +0000 UTC m=+44.708836982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:14.964356 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:14.964253    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s"]
Jul 27 20:52:14.967636 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:14.967609    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c93b3d_1e00_489a_9682_f72c95ac215c.slice/crio-2f79bd853b6754b3a236829f89285ed561631d023cbd9ffe2a72269a4e83817c WatchSource:0}: Error finding container 2f79bd853b6754b3a236829f89285ed561631d023cbd9ffe2a72269a4e83817c: Status 404 returned error can't find the container with id 2f79bd853b6754b3a236829f89285ed561631d023cbd9ffe2a72269a4e83817c
Jul 27 20:52:15.035066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.035032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:15.035066 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.035081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:15.035313 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:15.035187    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:15.035313 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:15.035195    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:16.035181616 +0000 UTC m=+41.809114014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:15.035313 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:15.035279    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:16.035266039 +0000 UTC m=+41.809198442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:15.135749 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.135565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:15.135749 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:15.135663    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 20:52:15.135749 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:15.135740    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls podName:e5f267ae-85e1-430e-84ff-c99b0c4b5d0a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:16.135721284 +0000 UTC m=+41.909653696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls") pod "cluster-samples-operator-6748489656-9g5vs" (UID: "e5f267ae-85e1-430e-84ff-c99b0c4b5d0a") : secret "samples-operator-tls" not found
Jul 27 20:52:15.195179 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.195142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s" event={"ID":"e2c93b3d-1e00-489a-9682-f72c95ac215c","Type":"ContainerStarted","Data":"2f79bd853b6754b3a236829f89285ed561631d023cbd9ffe2a72269a4e83817c"}
Jul 27 20:52:15.196543 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.196517    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m" event={"ID":"68d27cc1-738e-4060-a482-70dac6ca1c6a","Type":"ContainerStarted","Data":"048738f228768e988ef5337b5b0b6e05c9eff6c77d8e7d84a951a45c0b40ec98"}
Jul 27 20:52:15.198090 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:15.198031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z" event={"ID":"7c2bdc7d-2972-4eb0-a771-1858fa4d313b","Type":"ContainerStarted","Data":"7ecc21b22db28ef13808d447ef9f8520fdfdcaa8efafa2955f86ee2c2b042053"}
Jul 27 20:52:16.044891 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:16.044843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:16.045150 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:16.044917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:16.045150 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:16.045011    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:16.045150 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:16.045086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:18.045064712 +0000 UTC m=+43.818997111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:16.045150 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:16.045131    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:18.04511523 +0000 UTC m=+43.819047633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:16.146532 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:16.146488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:16.146701 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:16.146643    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 20:52:16.146777 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:16.146715    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls podName:e5f267ae-85e1-430e-84ff-c99b0c4b5d0a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:18.146694997 +0000 UTC m=+43.920627410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls") pod "cluster-samples-operator-6748489656-9g5vs" (UID: "e5f267ae-85e1-430e-84ff-c99b0c4b5d0a") : secret "samples-operator-tls" not found
Jul 27 20:52:18.063371 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.063330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:18.063858 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.063396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:18.063858 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.063504    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:18.063858 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.063581    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.063561636 +0000 UTC m=+47.837494048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:18.063858 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.063599    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.0635906 +0000 UTC m=+47.837522998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:18.164357 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.164311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:18.164557 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.164496    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 20:52:18.164610 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.164581    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls podName:e5f267ae-85e1-430e-84ff-c99b0c4b5d0a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.164559422 +0000 UTC m=+47.938491827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls") pod "cluster-samples-operator-6748489656-9g5vs" (UID: "e5f267ae-85e1-430e-84ff-c99b0c4b5d0a") : secret "samples-operator-tls" not found
Jul 27 20:52:18.578083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.578045    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"]
Jul 27 20:52:18.579850 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.579826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.590828 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.590807    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"
Jul 27 20:52:18.593000 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.592977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"]
Jul 27 20:52:18.611392 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.607823    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:18.611392 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.608039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-z2dvx\""
Jul 27 20:52:18.611392 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.610257    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 20:52:18.662261 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.662207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.669051 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.669025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7k2\" (UniqueName: \"kubernetes.io/projected/047c7c11-68fb-426e-9ed3-56de63b6b1c7-kube-api-access-xj7k2\") pod \"migrator-6865c9967-9gsnq\" (UID: \"047c7c11-68fb-426e-9ed3-56de63b6b1c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"
Jul 27 20:52:18.770442 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.770412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7k2\" (UniqueName: \"kubernetes.io/projected/047c7c11-68fb-426e-9ed3-56de63b6b1c7-kube-api-access-xj7k2\") pod \"migrator-6865c9967-9gsnq\" (UID: \"047c7c11-68fb-426e-9ed3-56de63b6b1c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"
Jul 27 20:52:18.778885 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.778861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7k2\" (UniqueName: \"kubernetes.io/projected/047c7c11-68fb-426e-9ed3-56de63b6b1c7-kube-api-access-xj7k2\") pod \"migrator-6865c9967-9gsnq\" (UID: \"047c7c11-68fb-426e-9ed3-56de63b6b1c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"
Jul 27 20:52:18.924714 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.924616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"
Jul 27 20:52:18.972844 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:18.972810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:18.973075 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.972994    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:18.973075 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:18.973054    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:26.973038093 +0000 UTC m=+52.746970508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:19.065549 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.065517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq"]
Jul 27 20:52:19.070302 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:19.070264    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047c7c11_68fb_426e_9ed3_56de63b6b1c7.slice/crio-43698b5a4ca4427c3746fdd30d1d5ccb945520871bde6a6ce9374045f0f4474b WatchSource:0}: Error finding container 43698b5a4ca4427c3746fdd30d1d5ccb945520871bde6a6ce9374045f0f4474b: Status 404 returned error can't find the container with id 43698b5a4ca4427c3746fdd30d1d5ccb945520871bde6a6ce9374045f0f4474b
Jul 27 20:52:19.136571 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.136531    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7xpr8"]
Jul 27 20:52:19.169915 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.169882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xpr8"]
Jul 27 20:52:19.170090 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.170035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.173694 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.173457    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:52:19.173694 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.173518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rxbs\""
Jul 27 20:52:19.173694 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.173459    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:52:19.173694 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.173460    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:52:19.174207 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.173847    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:52:19.179372 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.179343    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.213368 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.213330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s" event={"ID":"e2c93b3d-1e00-489a-9682-f72c95ac215c","Type":"ContainerStarted","Data":"edf6d9809fac28edf20c38dc07492fee9c65d3b268b0c67bb29b9dbdf9569bf7"}
Jul 27 20:52:19.214505 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.214472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq" event={"ID":"047c7c11-68fb-426e-9ed3-56de63b6b1c7","Type":"ContainerStarted","Data":"43698b5a4ca4427c3746fdd30d1d5ccb945520871bde6a6ce9374045f0f4474b"}
Jul 27 20:52:19.215993 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.215970    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m" event={"ID":"68d27cc1-738e-4060-a482-70dac6ca1c6a","Type":"ContainerStarted","Data":"c00afe7a2d94c479a396c3909546b87d34a44652143d290e7a37219623c41bd0"}
Jul 27 20:52:19.217625 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.217604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z" event={"ID":"7c2bdc7d-2972-4eb0-a771-1858fa4d313b","Type":"ContainerStarted","Data":"922783442cc09fd1ac11396bafd45bedbf311419f27568aca5dd8c169cda14c7"}
Jul 27 20:52:19.230455 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.230404    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-98m5s" podStartSLOduration=1.570177634 podStartE2EDuration="5.230388822s" totalImagesPullingTime="3.660211188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:19.229406246 +0000 UTC m=+45.003338667" watchObservedRunningTime="2026-07-27 20:52:19.230388822 +0000 UTC m=+45.004321243"
Jul 27 20:52:19.246766 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.246716    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-thw6z" podStartSLOduration=1.32673649 podStartE2EDuration="5.246700358s" totalImagesPullingTime="3.919963868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:19.245288424 +0000 UTC m=+45.019220849" watchObservedRunningTime="2026-07-27 20:52:19.246700358 +0000 UTC m=+45.020632778"
Jul 27 20:52:19.250584 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.250551    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.274243 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.274196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.274407 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.274381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.274498 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.274474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcrl\" (UniqueName: \"kubernetes.io/projected/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-api-access-clcrl\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.274588 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.274568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-data-volume\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.274700 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.274611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-crio-socket\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.322581 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.322544    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.375855 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.375810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-data-volume\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376029 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.375867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-crio-socket\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376029 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.375909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376029 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.375984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376168 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.376045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcrl\" (UniqueName: \"kubernetes.io/projected/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-api-access-clcrl\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376168 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.376068    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.376168 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.376136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-crio-socket\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376307 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.376169    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:19.876146505 +0000 UTC m=+45.650078922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.376307 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.376176    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.376307 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.376193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-data-volume\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.376307 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.376260    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:19.876242799 +0000 UTC m=+45.650175197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.384826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.384802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcrl\" (UniqueName: \"kubernetes.io/projected/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-api-access-clcrl\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.415023 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.414989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.880184 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.880141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.880386 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:19.880242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:19.880386 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.880295    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.880386 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.880378    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:20.880357118 +0000 UTC m=+46.654289533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.880496 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.880388    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.880496 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:19.880446    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:20.880434348 +0000 UTC m=+46.654366745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:20.311974 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.311932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.410055 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.410021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.738042 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.738002    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.838275 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.838235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.888575 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.888550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:20.888664 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.888623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:20.888702 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:20.888666    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:20.888739 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:20.888727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.888707117 +0000 UTC m=+48.662639517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:20.889007 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:20.888986    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:20.889074 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:20.889060    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.889042016 +0000 UTC m=+48.662974440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:20.942370 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:20.942289    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:21.038941 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.038888    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:21.226332 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.226294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq" event={"ID":"047c7c11-68fb-426e-9ed3-56de63b6b1c7","Type":"ContainerStarted","Data":"42140e844a93e7707ebc21ebf0d3273533ec59f2766f9eb9f367aff04107822d"}
Jul 27 20:52:21.226511 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.226378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq" event={"ID":"047c7c11-68fb-426e-9ed3-56de63b6b1c7","Type":"ContainerStarted","Data":"efd0c4f99ffcf15a529fb92e66dd9ea8ee71bf850348dcc1b70ccc1296c370e3"}
Jul 27 20:52:21.242379 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.242330    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m" podStartSLOduration=3.453389579 podStartE2EDuration="7.242316753s" totalImagesPullingTime="3.788927174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:19.267893434 +0000 UTC m=+45.041825856" watchObservedRunningTime="2026-07-27 20:52:21.242316753 +0000 UTC m=+47.016249218"
Jul 27 20:52:21.242509 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.242416    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9gsnq" podStartSLOduration=1.624181486 podStartE2EDuration="3.24240983s" totalImagesPullingTime="1.618228344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:21.241427111 +0000 UTC m=+47.015359531" watchObservedRunningTime="2026-07-27 20:52:21.24240983 +0000 UTC m=+47.016342251"
Jul 27 20:52:21.329368 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.329340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:21.433556 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.433521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:21.911531 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:21.911498    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52wdx_43894920-1dc9-4687-ad29-f55409631161/dns-node-resolver/0.log"
Jul 27 20:52:22.097959 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.097922    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:22.098168 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.097979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:22.098168 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.098107    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:22.098168 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.098129    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:30.098116189 +0000 UTC m=+55.872048587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:22.098351 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.098170    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:30.098153091 +0000 UTC m=+55.872085505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:22.198833 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.198735    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:22.198978 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.198882    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 27 20:52:22.198978 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.198947    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls podName:e5f267ae-85e1-430e-84ff-c99b0c4b5d0a nodeName:}" failed. No retries permitted until 2026-07-27 20:52:30.198932123 +0000 UTC m=+55.972864521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls") pod "cluster-samples-operator-6748489656-9g5vs" (UID: "e5f267ae-85e1-430e-84ff-c99b0c4b5d0a") : secret "samples-operator-tls" not found
Jul 27 20:52:22.578761 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.578720    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:22.641027 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.640994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:22.904353 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.904250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:22.904353 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.904313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:22.904564 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.904402    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:22.904564 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.904405    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:22.904564 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.904466    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:26.904450468 +0000 UTC m=+52.678382867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:22.904564 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:22.904481    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:26.904473905 +0000 UTC m=+52.678406303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:22.911598 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:22.911572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnmb6_bab66c4e-4a49-4752-9050-e2a53b6f9813/node-ca/0.log"
Jul 27 20:52:23.611631 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:23.611597    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:23.712461 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:23.712426    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n62fk_f4618ee4-fea8-4ac0-865f-8bf5c5c72c68/serve-healthcheck-canary/0.log"
Jul 27 20:52:24.112317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:24.112286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9gsnq_047c7c11-68fb-426e-9ed3-56de63b6b1c7/migrator/0.log"
Jul 27 20:52:24.312804 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:24.312771    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9gsnq_047c7c11-68fb-426e-9ed3-56de63b6b1c7/graceful-termination/0.log"
Jul 27 20:52:26.230209 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.230172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.392968 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.392920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.492651 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.492570    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.565027 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.564969    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.640824 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.640791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.824140 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.824056    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.937547 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.937516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:26.937749 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.937602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:26.937749 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:26.937653    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:26.937749 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:26.937725    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:34.937704101 +0000 UTC m=+60.711636499 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:26.937749 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:26.937735    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:26.937934 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:26.937795    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:34.937776528 +0000 UTC m=+60.711708955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:26.938254 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:26.938208    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:27.038101 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:27.038075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:27.038306 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:27.038272    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:27.038375 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:27.038346    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls podName:8b629b7e-680e-4019-a64e-0dc46331d1b9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:43.038325058 +0000 UTC m=+68.812257467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls") pod "dns-default-5dzsn" (UID: "8b629b7e-680e-4019-a64e-0dc46331d1b9") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:27.038625 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:27.038600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:27.327436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:27.327398    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:28.318980 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:28.318939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:28.525014 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:28.524964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:28.925765 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:28.925693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:29.128662 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:29.128621    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:29.521935 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:29.521890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:29.926809 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:29.926725    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.047736 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.047703    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.161572 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.161532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:30.161774 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.161591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:30.161774 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:30.161723    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 20:52:30.161884 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:30.161785    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:46.161765155 +0000 UTC m=+71.935697552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : secret "router-metrics-certs-default" not found
Jul 27 20:52:30.161884 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:30.161815    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle podName:72c0faa1-cedb-4a19-838f-6e37a26255e3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:46.161804884 +0000 UTC m=+71.935737302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle") pod "router-default-7c6b4bf9c-jn462" (UID: "72c0faa1-cedb-4a19-838f-6e37a26255e3") : configmap references non-existent config key: service-ca.crt
Jul 27 20:52:30.263237 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.263184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:30.265972 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.265936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e5f267ae-85e1-430e-84ff-c99b0c4b5d0a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9g5vs\" (UID: \"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:30.315349 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.315307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"
Jul 27 20:52:30.341639 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.341590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.442864 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.442831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs"]
Jul 27 20:52:30.722588 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.722555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.920228 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.920174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.989419 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:30.989328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:31.129654 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:31.129610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:31.258054 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:31.257938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs" event={"ID":"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a","Type":"ContainerStarted","Data":"bd79b7f986742fd5fdfd38fe6bb0d513ee8292e0c3e29f32c8d5fd4fe88e601f"}
Jul 27 20:52:31.330126 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:31.330084    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:31.708043 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:31.708001    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.263387 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.263350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs" event={"ID":"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a","Type":"ContainerStarted","Data":"7e757bf742cd3baef98fe8c5d6efed90e235abb3299eb4d4c64ffe6fcfcf105f"}
Jul 27 20:52:32.263770 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.263412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs" event={"ID":"e5f267ae-85e1-430e-84ff-c99b0c4b5d0a","Type":"ContainerStarted","Data":"b2716a7bd64098d98c55e1c5a422a0e1304f182beee531b6521cd7b54e1b1d16"}
Jul 27 20:52:32.282840 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.282786    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9g5vs" podStartSLOduration=16.647810416 podStartE2EDuration="18.282771374s" totalImagesPullingTime="1.634960958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:32.282527402 +0000 UTC m=+58.056459824" watchObservedRunningTime="2026-07-27 20:52:32.282771374 +0000 UTC m=+58.056703793"
Jul 27 20:52:32.467559 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.467501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.569748 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.569709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.736187 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.736105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.942772 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:32.942729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.123880 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:33.123784    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.361613 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:33.361578    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.485184 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:33.485152    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.580841 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:33.580809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.045960 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:34.045918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.330627 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:34.330526    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.722964 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:34.722914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.924813 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:34.924754    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:35.005575 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:35.005491    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:35.005575 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:35.005564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:35.005746 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:35.005711    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:35.005798 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:35.005787    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls podName:d37b317e-a9ec-466f-ad1f-4c43d9ccff51 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:51.005765396 +0000 UTC m=+76.779697797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xpr8" (UID: "d37b317e-a9ec-466f-ad1f-4c43d9ccff51") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:35.006149 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:35.006131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:35.125794 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:35.125756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:35.537403 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:35.537355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:36.123656 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:36.123622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:36.170553 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:36.170513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xk7hl"
Jul 27 20:52:36.559076 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:36.559025    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:36.736820 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:36.736788    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:36.946623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:36.946530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.032965 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:37.032935    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.134283 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:37.134248    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.287376 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:37.287339    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.668031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:37.667931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:38.055638 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:38.055608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:38.139426 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:38.139385    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:38.463773 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:38.463743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:38.663641 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:38.663602    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:39.068060 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.068027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:39.294861 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.294828    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:39.544108 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.544077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:39.544285 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.544113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:39.544285 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.544163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:39.546938 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.546906    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:52:39.547086 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.546997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdj2\" (UniqueName: \"kubernetes.io/projected/85d66181-00fe-43e8-abca-193045d85495-kube-api-access-9hdj2\") pod \"network-check-target-pxcw2\" (UID: \"85d66181-00fe-43e8-abca-193045d85495\") " pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:39.547914 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.547898    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:52:39.557408 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.557376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bd91b7f1-6d7e-4cfd-a400-373508ac1c0c-original-pull-secret\") pod \"global-pull-secret-syncer-m582w\" (UID: \"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c\") " pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:39.557547 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.557425    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3401dd96-76a0-49f2-8934-ceed9c91835c-metrics-certs\") pod \"network-metrics-daemon-fd7cv\" (UID: \"3401dd96-76a0-49f2-8934-ceed9c91835c\") " pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:39.710300 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.710263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m582w"
Jul 27 20:52:39.718533 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.718505    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qtl47\""
Jul 27 20:52:39.721962 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.721932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvgk8\""
Jul 27 20:52:39.726400 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.726369    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:39.730392 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.730370    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fd7cv"
Jul 27 20:52:39.862949 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.862920    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m582w"]
Jul 27 20:52:39.887712 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.887637    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pxcw2"]
Jul 27 20:52:39.891152 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:39.891126    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d66181_00fe_43e8_abca_193045d85495.slice/crio-a1e9ff267b470a5b85f7adddc3f4873796738a98ef181c8752ce3be628864a5f WatchSource:0}: Error finding container a1e9ff267b470a5b85f7adddc3f4873796738a98ef181c8752ce3be628864a5f: Status 404 returned error can't find the container with id a1e9ff267b470a5b85f7adddc3f4873796738a98ef181c8752ce3be628864a5f
Jul 27 20:52:39.896585 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.896529    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:39.897187 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:39.897164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fd7cv"]
Jul 27 20:52:39.900924 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:39.900905    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3401dd96_76a0_49f2_8934_ceed9c91835c.slice/crio-c3fe0178c4f11de329fc0751d0bf8f28d10d027177764ca8ebe6e43e0ed62377 WatchSource:0}: Error finding container c3fe0178c4f11de329fc0751d0bf8f28d10d027177764ca8ebe6e43e0ed62377: Status 404 returned error can't find the container with id c3fe0178c4f11de329fc0751d0bf8f28d10d027177764ca8ebe6e43e0ed62377
Jul 27 20:52:40.288427 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.288398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pxcw2" event={"ID":"85d66181-00fe-43e8-abca-193045d85495","Type":"ContainerStarted","Data":"9c58da6e0ccb84ef450e83e012f22bcb3abde658908b77a56226c424845b5fb5"}
Jul 27 20:52:40.288619 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.288443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pxcw2" event={"ID":"85d66181-00fe-43e8-abca-193045d85495","Type":"ContainerStarted","Data":"a1e9ff267b470a5b85f7adddc3f4873796738a98ef181c8752ce3be628864a5f"}
Jul 27 20:52:40.288619 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.288507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:52:40.289618 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.289596    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd7cv" event={"ID":"3401dd96-76a0-49f2-8934-ceed9c91835c","Type":"ContainerStarted","Data":"c3fe0178c4f11de329fc0751d0bf8f28d10d027177764ca8ebe6e43e0ed62377"}
Jul 27 20:52:40.290624 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.290605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m582w" event={"ID":"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c","Type":"ContainerStarted","Data":"457829b41f401cb5a65bcd5d2d63f0946c644539c033e1fc6093c8d5b57a28d4"}
Jul 27 20:52:40.305690 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:40.305651    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pxcw2" podStartSLOduration=65.305637374 podStartE2EDuration="1m5.305637374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:40.305064935 +0000 UTC m=+66.078997355" watchObservedRunningTime="2026-07-27 20:52:40.305637374 +0000 UTC m=+66.079569794"
Jul 27 20:52:41.296330 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:41.296250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd7cv" event={"ID":"3401dd96-76a0-49f2-8934-ceed9c91835c","Type":"ContainerStarted","Data":"5cfd6fd3ecd605b8e5a6d97d0b21e8133a26a4447c712bf152d525481821f3f9"}
Jul 27 20:52:41.296330 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:41.296313    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fd7cv" event={"ID":"3401dd96-76a0-49f2-8934-ceed9c91835c","Type":"ContainerStarted","Data":"34e2a76728348dd5ac63bcc2f1787fb50c1e2197d4eb4d6e4a74641e740c8baa"}
Jul 27 20:52:41.315386 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:41.315337    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fd7cv" podStartSLOduration=65.196283867 podStartE2EDuration="1m6.315321125s" totalImagesPullingTime="1.119037258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:41.313326372 +0000 UTC m=+67.087258794" watchObservedRunningTime="2026-07-27 20:52:41.315321125 +0000 UTC m=+67.089253546"
Jul 27 20:52:43.070829 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:43.070795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:43.073619 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:43.073595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b629b7e-680e-4019-a64e-0dc46331d1b9-metrics-tls\") pod \"dns-default-5dzsn\" (UID: \"8b629b7e-680e-4019-a64e-0dc46331d1b9\") " pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:43.267433 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:43.267393    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9tww\""
Jul 27 20:52:43.274637 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:43.274614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:43.963700 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:43.963676    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5dzsn"]
Jul 27 20:52:43.966097 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:43.966068    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b629b7e_680e_4019_a64e_0dc46331d1b9.slice/crio-7cd06a465fc77a67663ba0f7a9a00713fb8409ea25554063ac0986b06db4226e WatchSource:0}: Error finding container 7cd06a465fc77a67663ba0f7a9a00713fb8409ea25554063ac0986b06db4226e: Status 404 returned error can't find the container with id 7cd06a465fc77a67663ba0f7a9a00713fb8409ea25554063ac0986b06db4226e
Jul 27 20:52:44.307279 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:44.307251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dzsn" event={"ID":"8b629b7e-680e-4019-a64e-0dc46331d1b9","Type":"ContainerStarted","Data":"7cd06a465fc77a67663ba0f7a9a00713fb8409ea25554063ac0986b06db4226e"}
Jul 27 20:52:45.313092 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:45.313047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m582w" event={"ID":"bd91b7f1-6d7e-4cfd-a400-373508ac1c0c","Type":"ContainerStarted","Data":"b6a1c84cdd92cad7a5442ab7688129abf287539efdfe23ba12af6acad492d6cb"}
Jul 27 20:52:45.329655 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:45.329606    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m582w" podStartSLOduration=66.013618588 podStartE2EDuration="1m10.329585509s" totalImagesPullingTime="4.315966921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:45.328786258 +0000 UTC m=+71.102718680" watchObservedRunningTime="2026-07-27 20:52:45.329585509 +0000 UTC m=+71.103517929"
Jul 27 20:52:46.086599 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.086561    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:46.187705 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.187668    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:46.198646 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.198624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:46.198720 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.198662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:46.199260 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.199233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72c0faa1-cedb-4a19-838f-6e37a26255e3-service-ca-bundle\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:46.201040 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.201023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72c0faa1-cedb-4a19-838f-6e37a26255e3-metrics-certs\") pod \"router-default-7c6b4bf9c-jn462\" (UID: \"72c0faa1-cedb-4a19-838f-6e37a26255e3\") " pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:46.317825 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.317787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dzsn" event={"ID":"8b629b7e-680e-4019-a64e-0dc46331d1b9","Type":"ContainerStarted","Data":"b8c742c785f02ffb7dbe711d5eaf64f8d30890560bb038994bd193c71424ccfc"}
Jul 27 20:52:46.318321 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.317850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5dzsn" event={"ID":"8b629b7e-680e-4019-a64e-0dc46331d1b9","Type":"ContainerStarted","Data":"62d24f7a2f34310425f8f6c211772b32769468dbe8692c3b926f5520ca82e061"}
Jul 27 20:52:46.338014 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.337921    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5dzsn" podStartSLOduration=33.745526986 podStartE2EDuration="35.337906033s" totalImagesPullingTime="1.592379047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:46.336378171 +0000 UTC m=+72.110310590" watchObservedRunningTime="2026-07-27 20:52:46.337906033 +0000 UTC m=+72.111838452"
Jul 27 20:52:46.401044 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.401017    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vf267\""
Jul 27 20:52:46.409416 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.409398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:46.539135 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:46.539087    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c6b4bf9c-jn462"]
Jul 27 20:52:46.543282 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:46.543258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c0faa1_cedb_4a19_838f_6e37a26255e3.slice/crio-89d3de4a52d3db2e88694892d8024018061b6b6be11ed7baacc07b634d9980dc WatchSource:0}: Error finding container 89d3de4a52d3db2e88694892d8024018061b6b6be11ed7baacc07b634d9980dc: Status 404 returned error can't find the container with id 89d3de4a52d3db2e88694892d8024018061b6b6be11ed7baacc07b634d9980dc
Jul 27 20:52:47.323033 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.322994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c6b4bf9c-jn462" event={"ID":"72c0faa1-cedb-4a19-838f-6e37a26255e3","Type":"ContainerStarted","Data":"a11c3e1791dc4499c41fdd63eaebf0a0644dd510556eca5e66c1b959140ce97b"}
Jul 27 20:52:47.323620 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.323048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c6b4bf9c-jn462" event={"ID":"72c0faa1-cedb-4a19-838f-6e37a26255e3","Type":"ContainerStarted","Data":"89d3de4a52d3db2e88694892d8024018061b6b6be11ed7baacc07b634d9980dc"}
Jul 27 20:52:47.323620 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.323264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:47.346556 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.346510    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7c6b4bf9c-jn462" podStartSLOduration=33.34649665 podStartE2EDuration="33.34649665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:47.345734819 +0000 UTC m=+73.119667238" watchObservedRunningTime="2026-07-27 20:52:47.34649665 +0000 UTC m=+73.120429070"
Jul 27 20:52:47.409776 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.409752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:47.412208 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.412188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:47.899739 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.899705    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-v6pj8"]
Jul 27 20:52:47.901601 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.901586    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:52:47.905586 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.905564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xrsjc\""
Jul 27 20:52:47.905781 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.905762    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:52:47.906177 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.906159    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:52:47.933597 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:47.933566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-v6pj8"]
Jul 27 20:52:48.008648 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.008615    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659bfb4c76-tld5w"]
Jul 27 20:52:48.010737 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.010718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.012982 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.012952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqv2\" (UniqueName: \"kubernetes.io/projected/719c5e99-4b9e-4217-af88-fa2abfc025ef-kube-api-access-fdqv2\") pod \"downloads-7c97dc4555-v6pj8\" (UID: \"719c5e99-4b9e-4217-af88-fa2abfc025ef\") " pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:52:48.016544 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.016521    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 20:52:48.016687 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.016659    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 20:52:48.017830 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.017810    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jv4bm\""
Jul 27 20:52:48.017991 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.017977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 20:52:48.023258 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.023239    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 20:52:48.030578 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.030558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659bfb4c76-tld5w"]
Jul 27 20:52:48.091085 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.091050    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.113939 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.113912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-image-registry-private-configuration\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114077 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.113959    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-tls\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114077 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-installation-pull-secrets\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114077 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-certificates\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114176 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lmz\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-kube-api-access-97lmz\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114176 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4257a616-d994-4f32-8cc9-6efa447aa1a1-ca-trust-extracted\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114176 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-trusted-ca\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114304 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-bound-sa-token\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.114304 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.114252    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqv2\" (UniqueName: \"kubernetes.io/projected/719c5e99-4b9e-4217-af88-fa2abfc025ef-kube-api-access-fdqv2\") pod \"downloads-7c97dc4555-v6pj8\" (UID: \"719c5e99-4b9e-4217-af88-fa2abfc025ef\") " pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:52:48.127287 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.127259    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqv2\" (UniqueName: \"kubernetes.io/projected/719c5e99-4b9e-4217-af88-fa2abfc025ef-kube-api-access-fdqv2\") pod \"downloads-7c97dc4555-v6pj8\" (UID: \"719c5e99-4b9e-4217-af88-fa2abfc025ef\") " pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:52:48.187278 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.187193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.210852 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.210826    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:52:48.214929 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.214909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-installation-pull-secrets\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215032 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.214945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-certificates\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215032 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.214972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lmz\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-kube-api-access-97lmz\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215032 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4257a616-d994-4f32-8cc9-6efa447aa1a1-ca-trust-extracted\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215177 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-trusted-ca\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215177 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-bound-sa-token\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215303 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215231    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-image-registry-private-configuration\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215303 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-tls\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4257a616-d994-4f32-8cc9-6efa447aa1a1-ca-trust-extracted\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.215967 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.215942    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-certificates\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.216604 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.216583    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4257a616-d994-4f32-8cc9-6efa447aa1a1-trusted-ca\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.217932 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.217912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-installation-pull-secrets\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.218333 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.218315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-registry-tls\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.218457 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.218435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4257a616-d994-4f32-8cc9-6efa447aa1a1-image-registry-private-configuration\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.232772 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.232744    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-bound-sa-token\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.234349 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.234327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lmz\" (UniqueName: \"kubernetes.io/projected/4257a616-d994-4f32-8cc9-6efa447aa1a1-kube-api-access-97lmz\") pod \"image-registry-659bfb4c76-tld5w\" (UID: \"4257a616-d994-4f32-8cc9-6efa447aa1a1\") " pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.314180 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.314145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.320102 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.320076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:48.326275 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.326246    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:48.327521 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.327498    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7c6b4bf9c-jn462"
Jul 27 20:52:48.346800 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.346729    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-v6pj8"]
Jul 27 20:52:48.350184 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:48.350161    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c5e99_4b9e_4217_af88_fa2abfc025ef.slice/crio-43d4f5b521631eb79d678c29ed7e2f64f4e36fcacda035b99ee5f6c5467ae234 WatchSource:0}: Error finding container 43d4f5b521631eb79d678c29ed7e2f64f4e36fcacda035b99ee5f6c5467ae234: Status 404 returned error can't find the container with id 43d4f5b521631eb79d678c29ed7e2f64f4e36fcacda035b99ee5f6c5467ae234
Jul 27 20:52:48.468358 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.468326    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.503240 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.503194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659bfb4c76-tld5w"]
Jul 27 20:52:48.506256 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:48.506200    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4257a616_d994_4f32_8cc9_6efa447aa1a1.slice/crio-dd437f611e43afc7a654f643c6410a15d047df4a6575a3c078a699f521266e20 WatchSource:0}: Error finding container dd437f611e43afc7a654f643c6410a15d047df4a6575a3c078a699f521266e20: Status 404 returned error can't find the container with id dd437f611e43afc7a654f643c6410a15d047df4a6575a3c078a699f521266e20
Jul 27 20:52:48.760554 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.760515    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.876671 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.876615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.980590 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.980556    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"]
Jul 27 20:52:48.982671 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.982649    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:48.985418 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.985395    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tnbbm\""
Jul 27 20:52:48.985662 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.985647    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 20:52:48.988597 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.988505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.995893 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:48.995868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"]
Jul 27 20:52:49.121766 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.121684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q2qcp\" (UID: \"39b51e8b-e3ea-46c8-a564-e8435cee15b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:49.222261 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.222200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q2qcp\" (UID: \"39b51e8b-e3ea-46c8-a564-e8435cee15b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:49.222485 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:49.222464    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:52:49.222570 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:49.222545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates podName:39b51e8b-e3ea-46c8-a564-e8435cee15b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:49.722523546 +0000 UTC m=+75.496455960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-q2qcp" (UID: "39b51e8b-e3ea-46c8-a564-e8435cee15b4") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:52:49.223236 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.223187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:49.331827 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.331790    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w" event={"ID":"4257a616-d994-4f32-8cc9-6efa447aa1a1","Type":"ContainerStarted","Data":"f3c9d402f7331fcf338a2cc10b82cc90e334230c64ecb7beed6d69a4725cc187"}
Jul 27 20:52:49.332321 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.331848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w" event={"ID":"4257a616-d994-4f32-8cc9-6efa447aa1a1","Type":"ContainerStarted","Data":"dd437f611e43afc7a654f643c6410a15d047df4a6575a3c078a699f521266e20"}
Jul 27 20:52:49.332321 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.332002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:52:49.338499 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.338455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-v6pj8" event={"ID":"719c5e99-4b9e-4217-af88-fa2abfc025ef","Type":"ContainerStarted","Data":"43d4f5b521631eb79d678c29ed7e2f64f4e36fcacda035b99ee5f6c5467ae234"}
Jul 27 20:52:49.355196 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.355137    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w" podStartSLOduration=2.3551194 podStartE2EDuration="2.3551194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:49.354110232 +0000 UTC m=+75.128042653" watchObservedRunningTime="2026-07-27 20:52:49.3551194 +0000 UTC m=+75.129051820"
Jul 27 20:52:49.612810 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.612739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:49.714397 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.714353    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:49.727112 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.727070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q2qcp\" (UID: \"39b51e8b-e3ea-46c8-a564-e8435cee15b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:49.727323 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:49.727239    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:52:49.727323 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:49.727309    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates podName:39b51e8b-e3ea-46c8-a564-e8435cee15b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:50.72729316 +0000 UTC m=+76.501225557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-q2qcp" (UID: "39b51e8b-e3ea-46c8-a564-e8435cee15b4") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 20:52:49.997050 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:49.997006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:50.737188 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:50.737149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q2qcp\" (UID: \"39b51e8b-e3ea-46c8-a564-e8435cee15b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:50.739954 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:50.739924    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/39b51e8b-e3ea-46c8-a564-e8435cee15b4-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-q2qcp\" (UID: \"39b51e8b-e3ea-46c8-a564-e8435cee15b4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:50.793735 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:50.793694    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:50.961379 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:50.961342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"]
Jul 27 20:52:50.964561 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:50.964517    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b51e8b_e3ea_46c8_a564_e8435cee15b4.slice/crio-f0e7cf9e5bbe83a12c8e18b79d0fbadba4f753c56f659c0ea19d50774b4cc411 WatchSource:0}: Error finding container f0e7cf9e5bbe83a12c8e18b79d0fbadba4f753c56f659c0ea19d50774b4cc411: Status 404 returned error can't find the container with id f0e7cf9e5bbe83a12c8e18b79d0fbadba4f753c56f659c0ea19d50774b4cc411
Jul 27 20:52:51.039840 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.039741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:51.042337 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.042312    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d37b317e-a9ec-466f-ad1f-4c43d9ccff51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xpr8\" (UID: \"d37b317e-a9ec-466f-ad1f-4c43d9ccff51\") " pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:51.284068 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.284034    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rxbs\""
Jul 27 20:52:51.291395 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.291323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xpr8"
Jul 27 20:52:51.347839 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.347802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp" event={"ID":"39b51e8b-e3ea-46c8-a564-e8435cee15b4","Type":"ContainerStarted","Data":"f0e7cf9e5bbe83a12c8e18b79d0fbadba4f753c56f659c0ea19d50774b4cc411"}
Jul 27 20:52:51.471304 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:51.471264    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xpr8"]
Jul 27 20:52:51.477349 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:51.477309    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37b317e_a9ec_466f_ad1f_4c43d9ccff51.slice/crio-47a78ad44fa26da68f0a30c42194110a58c81f975f645a899792a33be24ebe14 WatchSource:0}: Error finding container 47a78ad44fa26da68f0a30c42194110a58c81f975f645a899792a33be24ebe14: Status 404 returned error can't find the container with id 47a78ad44fa26da68f0a30c42194110a58c81f975f645a899792a33be24ebe14
Jul 27 20:52:52.353631 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:52.353590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xpr8" event={"ID":"d37b317e-a9ec-466f-ad1f-4c43d9ccff51","Type":"ContainerStarted","Data":"47a78ad44fa26da68f0a30c42194110a58c81f975f645a899792a33be24ebe14"}
Jul 27 20:52:53.360352 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:53.360308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp" event={"ID":"39b51e8b-e3ea-46c8-a564-e8435cee15b4","Type":"ContainerStarted","Data":"3bdccaef812bfaef2d22d1f74b03023941bf2670dbfa8d7914dbde10cdfd434b"}
Jul 27 20:52:53.360896 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:53.360874    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:53.367198 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:53.367166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp"
Jul 27 20:52:53.382466 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:53.382397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-q2qcp" podStartSLOduration=3.862179153 podStartE2EDuration="5.38237669s" totalImagesPullingTime="1.520197537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:53.381260315 +0000 UTC m=+79.155192736" watchObservedRunningTime="2026-07-27 20:52:53.38237669 +0000 UTC m=+79.156309120"
Jul 27 20:52:53.997240 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:53.997177    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.064430 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.063115    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"]
Jul 27 20:52:54.066368 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.066336    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.068848 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.068823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 20:52:54.070231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.070194    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:52:54.070363 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.070291    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8kb62\""
Jul 27 20:52:54.070363 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.070347    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:52:54.070480 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.070295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 20:52:54.070527 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.070492    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:52:54.080605 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.080580    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"]
Jul 27 20:52:54.094187 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.094102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.168574 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.168527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.168754 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.168589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea68fd-2947-4918-bb1a-7b4b4669b768-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.168754 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.168671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.168754 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.168720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnjtg\" (UniqueName: \"kubernetes.io/projected/d2ea68fd-2947-4918-bb1a-7b4b4669b768-kube-api-access-gnjtg\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.269842 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.269754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea68fd-2947-4918-bb1a-7b4b4669b768-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.269842 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.269813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.270083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.269850    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnjtg\" (UniqueName: \"kubernetes.io/projected/d2ea68fd-2947-4918-bb1a-7b4b4669b768-kube-api-access-gnjtg\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.270083 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.269924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.270545 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:54.270524    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 20:52:54.270615 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:52:54.270608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls podName:d2ea68fd-2947-4918-bb1a-7b4b4669b768 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:54.770584583 +0000 UTC m=+80.544516984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-kjhrt" (UID: "d2ea68fd-2947-4918-bb1a-7b4b4669b768") : secret "prometheus-operator-tls" not found
Jul 27 20:52:54.271572 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.271545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2ea68fd-2947-4918-bb1a-7b4b4669b768-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.273060 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.273033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.281466 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.281434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnjtg\" (UniqueName: \"kubernetes.io/projected/d2ea68fd-2947-4918-bb1a-7b4b4669b768-kube-api-access-gnjtg\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.412779 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.412733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.654024 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.653936    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.774121 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.773970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.777737 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.777175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2ea68fd-2947-4918-bb1a-7b4b4669b768-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kjhrt\" (UID: \"d2ea68fd-2947-4918-bb1a-7b4b4669b768\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:54.826286 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.826250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.920461 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.920376    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.979399 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:54.979354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"
Jul 27 20:52:55.542427 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:55.542355    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:55.642738 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:55.642686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:57.328510 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:57.328473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5dzsn"
Jul 27 20:52:57.814472 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:57.814443    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kjhrt"]
Jul 27 20:52:57.816787 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:52:57.816748    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ea68fd_2947_4918_bb1a_7b4b4669b768.slice/crio-423e2423e1b5e41277a7b699088ff1ed70cf9e1f641d7f7d80812d08b3eab148 WatchSource:0}: Error finding container 423e2423e1b5e41277a7b699088ff1ed70cf9e1f641d7f7d80812d08b3eab148: Status 404 returned error can't find the container with id 423e2423e1b5e41277a7b699088ff1ed70cf9e1f641d7f7d80812d08b3eab148
Jul 27 20:52:58.385299 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:58.385169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt" event={"ID":"d2ea68fd-2947-4918-bb1a-7b4b4669b768","Type":"ContainerStarted","Data":"423e2423e1b5e41277a7b699088ff1ed70cf9e1f641d7f7d80812d08b3eab148"}
Jul 27 20:52:58.388521 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:58.388401    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xpr8" event={"ID":"d37b317e-a9ec-466f-ad1f-4c43d9ccff51","Type":"ContainerStarted","Data":"e20b04e1cf6c746efa0cae77ab41a003b4d728d82fdfbbf73656762c653b6bf0"}
Jul 27 20:52:59.779577 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:52:59.779536    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:00.399600 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:00.399555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt" event={"ID":"d2ea68fd-2947-4918-bb1a-7b4b4669b768","Type":"ContainerStarted","Data":"08f74716f4e4a8745e56366b16fb4da963457b48804abf722ade4dd4454f3422"}
Jul 27 20:53:00.399757 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:00.399626    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt" event={"ID":"d2ea68fd-2947-4918-bb1a-7b4b4669b768","Type":"ContainerStarted","Data":"64b2bb7fcea91491848f3235309cf7b172e8774dab0c93440e488141b33b60e2"}
Jul 27 20:53:00.429358 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:00.429297    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-kjhrt" podStartSLOduration=4.875373838 podStartE2EDuration="6.429276086s" totalImagesPullingTime="1.553902248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:00.427307651 +0000 UTC m=+86.201240075" watchObservedRunningTime="2026-07-27 20:53:00.429276086 +0000 UTC m=+86.203208501"
Jul 27 20:53:02.212599 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.212562    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.405707 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.405657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.436369 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.436334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"]
Jul 27 20:53:02.439799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.439669    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.442764 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.442639    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 20:53:02.443116 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.442914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 20:53:02.443116 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.443067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vhh48\""
Jul 27 20:53:02.453399 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.453369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"]
Jul 27 20:53:02.496317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.496224    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2pzqf"]
Jul 27 20:53:02.498986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.498954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.501616 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.501589    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwms7\""
Jul 27 20:53:02.501978 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.501684    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:53:02.502196 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.501684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:53:02.502440 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.502419    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:53:02.515042 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.515006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.543045 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.543009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a23771-4518-48de-95d5-92b93d00d0a3-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.543045 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.543050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.543345 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.543116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.543345 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.543144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686tv\" (UniqueName: \"kubernetes.io/projected/63a23771-4518-48de-95d5-92b93d00d0a3-kube-api-access-686tv\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.608754 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.608657    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.644589 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.644782 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686tv\" (UniqueName: \"kubernetes.io/projected/63a23771-4518-48de-95d5-92b93d00d0a3-kube-api-access-686tv\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.644847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-textfile\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.644847 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-metrics-client-ca\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.644952 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645003 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-tls\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645003 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.644984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-sys\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645104 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a23771-4518-48de-95d5-92b93d00d0a3-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.645104 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.645104 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645088    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645291 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-wtmp\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645291 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/f0fc842b-d61e-47d4-9630-b9cd0ba04117-kube-api-access-lwt7j\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645291 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-root\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.645906 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.645880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63a23771-4518-48de-95d5-92b93d00d0a3-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.647951 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.647928    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.648112 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.647967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63a23771-4518-48de-95d5-92b93d00d0a3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.654458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.654396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-686tv\" (UniqueName: \"kubernetes.io/projected/63a23771-4518-48de-95d5-92b93d00d0a3-kube-api-access-686tv\") pod \"openshift-state-metrics-b8f49f5c6-m69rm\" (UID: \"63a23771-4518-48de-95d5-92b93d00d0a3\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.708109 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.708072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.746471 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-textfile\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746471 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-metrics-client-ca\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746471 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746727 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-tls\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746784 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-sys\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746834 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746834 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-wtmp\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746933 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746857    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/f0fc842b-d61e-47d4-9630-b9cd0ba04117-kube-api-access-lwt7j\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.746933 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-root\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747023 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.746828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-textfile\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747023 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.747003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-sys\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747120 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.747049    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-root\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747173 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.747151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-wtmp\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747502 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.747483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.747572 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.747523    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0fc842b-d61e-47d4-9630-b9cd0ba04117-metrics-client-ca\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.749771 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.749748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.749771 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.749765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0fc842b-d61e-47d4-9630-b9cd0ba04117-node-exporter-tls\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.756924 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.756689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"
Jul 27 20:53:02.767186 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.767159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwt7j\" (UniqueName: \"kubernetes.io/projected/f0fc842b-d61e-47d4-9630-b9cd0ba04117-kube-api-access-lwt7j\") pod \"node-exporter-2pzqf\" (UID: \"f0fc842b-d61e-47d4-9630-b9cd0ba04117\") " pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.810469 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.810437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2pzqf"
Jul 27 20:53:02.811199 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.811163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.912033 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:02.911892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.013573 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.013477    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.348780 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.348681    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.448433 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.448390    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.553300 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.553234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.649846 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.649752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.776536 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:03.776492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:04.616181 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:04.616074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:04.716940 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:04.716898    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:05.494555 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:05.494500    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:05.605703 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:05.605666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:05.708065 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:05.708023    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:05.797609 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:05.797521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:06.496469 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:06.496428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:06.562563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:06.562523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:07.236947 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.236889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:07.274317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.274276    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"]
Jul 27 20:53:07.301193 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.301155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:53:07.301372 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.301318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.304068 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.303838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 20:53:07.304232 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.304194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pph7w\""
Jul 27 20:53:07.321717 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.321674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:07.325960 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.325930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"]
Jul 27 20:53:07.325960 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.325965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:53:07.326168 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.326054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.328925 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.328592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:53:07.328925 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.328748    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:53:07.328925 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.328872    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:53:07.328925 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.328909    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nr66h\""
Jul 27 20:53:07.328925 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.328908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:53:07.329335 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.329193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:53:07.334390 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.334361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:53:07.490690 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.490690 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.490890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/251ecbce-92e2-486d-8292-8f593eeac49a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.490890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dml67\" (UniqueName: \"kubernetes.io/projected/251ecbce-92e2-486d-8292-8f593eeac49a-kube-api-access-dml67\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.490890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.490890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzr9m\" (UniqueName: \"kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.490890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.491088 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.490906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.491088 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.491001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.591788 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.591753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzr9m\" (UniqueName: \"kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592001 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.591808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592054 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.591996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592087 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592126 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592126 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592891 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.592975 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/251ecbce-92e2-486d-8292-8f593eeac49a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.592975 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.593046 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dml67\" (UniqueName: \"kubernetes.io/projected/251ecbce-92e2-486d-8292-8f593eeac49a-kube-api-access-dml67\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.593046 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.592984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.593046 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.593005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.593702 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.593619    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.594923 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.594891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.595715 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.595691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/251ecbce-92e2-486d-8292-8f593eeac49a-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.595715 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.595710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.601119 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.601094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzr9m\" (UniqueName: \"kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m\") pod \"console-5dcb746989-x9mz2\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") " pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:07.601883 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.601860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dml67\" (UniqueName: \"kubernetes.io/projected/251ecbce-92e2-486d-8292-8f593eeac49a-kube-api-access-dml67\") pod \"monitoring-plugin-68b7fc9574-prdpn\" (UID: \"251ecbce-92e2-486d-8292-8f593eeac49a\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.609845 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.609816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:07.613934 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.613704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:07.637748 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:07.637715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:08.779005 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.778974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:08.788448 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.788411    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:53:08.885984 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.885946    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.890802 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.890773    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:53:08.891189 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.891067    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893104    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893174    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82gkd50gr60i6\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893381    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893565    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893600    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:53:08.893734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893657    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j724c\""
Jul 27 20:53:08.894294 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.893880    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:53:08.894351 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.894328    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:53:08.895185 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.894574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:53:08.895185 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.894667    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:53:08.895185 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.895010    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:53:08.895185 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.895104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:53:08.897287 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.897264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:53:08.903155 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903128    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903307 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903186    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903360 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903403 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903537 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903574 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903613 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903763 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903743    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903820 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903820 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903898 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903937 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.903937 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904026 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.903955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904026 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.904004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904128 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.904059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904128 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.904091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904128 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.904119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:08.904275 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:08.904144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q466q\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004609 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.004799 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.004978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005093    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005136 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q466q\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.005460 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.005410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.006359 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.006330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.006582 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.006508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.006582 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.006527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.007611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.006934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.007611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.007021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.007611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.007465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.007914 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.007891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.008842 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.008817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.009167 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.009105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.011139 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.011107    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.011643 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.011573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.011643 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.011635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.011956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.011729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.012436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.012144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.012436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.012395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.012663 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.012639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.012801 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.012786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.013353 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.013333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.013909 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.013889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.017181 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.017154    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q466q\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q\") pod \"prometheus-k8s-0\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.201592 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.201182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:09.325416 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:53:09.324764    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fc842b_d61e_47d4_9630_b9cd0ba04117.slice/crio-904f27d0311693255ee7406f77d7200e44a1bf8dcd18294bc98745bdbe687433 WatchSource:0}: Error finding container 904f27d0311693255ee7406f77d7200e44a1bf8dcd18294bc98745bdbe687433: Status 404 returned error can't find the container with id 904f27d0311693255ee7406f77d7200e44a1bf8dcd18294bc98745bdbe687433
Jul 27 20:53:09.441551 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.441454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pzqf" event={"ID":"f0fc842b-d61e-47d4-9630-b9cd0ba04117","Type":"ContainerStarted","Data":"904f27d0311693255ee7406f77d7200e44a1bf8dcd18294bc98745bdbe687433"}
Jul 27 20:53:09.508639 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.507800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm"]
Jul 27 20:53:09.510957 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:53:09.510922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a23771_4518_48de_95d5_92b93d00d0a3.slice/crio-7e250196c993204f093de3db4858a6abd1d226edde6dd7bc635ac35b598dd9b4 WatchSource:0}: Error finding container 7e250196c993204f093de3db4858a6abd1d226edde6dd7bc635ac35b598dd9b4: Status 404 returned error can't find the container with id 7e250196c993204f093de3db4858a6abd1d226edde6dd7bc635ac35b598dd9b4
Jul 27 20:53:09.570418 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.570371    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:53:09.574698 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:53:09.574661    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac66374_5853_4ba9_9f81_247cbb0a3417.slice/crio-2f6e602f2ecc18ea8f7f1588f32e32a7dd9192b6fa11f01aaa67b60c2ae9584c WatchSource:0}: Error finding container 2f6e602f2ecc18ea8f7f1588f32e32a7dd9192b6fa11f01aaa67b60c2ae9584c: Status 404 returned error can't find the container with id 2f6e602f2ecc18ea8f7f1588f32e32a7dd9192b6fa11f01aaa67b60c2ae9584c
Jul 27 20:53:09.729380 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.729343    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"]
Jul 27 20:53:09.735836 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:53:09.735800    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251ecbce_92e2_486d_8292_8f593eeac49a.slice/crio-9c677d14545ca040e4ea5416234622ab68409488532def271f3a5e7d940a9003 WatchSource:0}: Error finding container 9c677d14545ca040e4ea5416234622ab68409488532def271f3a5e7d940a9003: Status 404 returned error can't find the container with id 9c677d14545ca040e4ea5416234622ab68409488532def271f3a5e7d940a9003
Jul 27 20:53:09.736748 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:09.736681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:53:09.747805 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:53:09.747769    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090cce42_479c_457c_90fc_b7b7b771e0a3.slice/crio-79d90ca4471ccf2343aeb913d108a2491ab326dcf898261aaf37e15ea2a610b4 WatchSource:0}: Error finding container 79d90ca4471ccf2343aeb913d108a2491ab326dcf898261aaf37e15ea2a610b4: Status 404 returned error can't find the container with id 79d90ca4471ccf2343aeb913d108a2491ab326dcf898261aaf37e15ea2a610b4
Jul 27 20:53:10.348590 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.348556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-659bfb4c76-tld5w"
Jul 27 20:53:10.454657 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.454613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-v6pj8" event={"ID":"719c5e99-4b9e-4217-af88-fa2abfc025ef","Type":"ContainerStarted","Data":"c65447c2c7a8c7f03a3656b21cfd53cd16b63857fd68bf17248f0ee22c10e0dc"}
Jul 27 20:53:10.457855 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.457829    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:53:10.460731 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.460710    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-v6pj8"
Jul 27 20:53:10.463843 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.463802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm" event={"ID":"63a23771-4518-48de-95d5-92b93d00d0a3","Type":"ContainerStarted","Data":"033f81a79e62769e9bb5c0aa110fe7d5ac3e7a9fc62de8addb52fac7f62a478e"}
Jul 27 20:53:10.464076 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.464049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm" event={"ID":"63a23771-4518-48de-95d5-92b93d00d0a3","Type":"ContainerStarted","Data":"136606d4d4a52fef416d9b804a4ef553a1097d35ed54d14139d485c9c64acd86"}
Jul 27 20:53:10.464194 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.464090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm" event={"ID":"63a23771-4518-48de-95d5-92b93d00d0a3","Type":"ContainerStarted","Data":"7e250196c993204f093de3db4858a6abd1d226edde6dd7bc635ac35b598dd9b4"}
Jul 27 20:53:10.470106 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.470067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn" event={"ID":"251ecbce-92e2-486d-8292-8f593eeac49a","Type":"ContainerStarted","Data":"9c677d14545ca040e4ea5416234622ab68409488532def271f3a5e7d940a9003"}
Jul 27 20:53:10.474181 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.474141    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"2f6e602f2ecc18ea8f7f1588f32e32a7dd9192b6fa11f01aaa67b60c2ae9584c"}
Jul 27 20:53:10.477263 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.477228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcb746989-x9mz2" event={"ID":"090cce42-479c-457c-90fc-b7b7b771e0a3","Type":"ContainerStarted","Data":"79d90ca4471ccf2343aeb913d108a2491ab326dcf898261aaf37e15ea2a610b4"}
Jul 27 20:53:10.484361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.484329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xpr8" event={"ID":"d37b317e-a9ec-466f-ad1f-4c43d9ccff51","Type":"ContainerStarted","Data":"341bab9ae991835254c856ee0205e8235b1df6eb40811809cec4921d84eecf94"}
Jul 27 20:53:10.561677 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.559897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-v6pj8" podStartSLOduration=2.538283152 podStartE2EDuration="23.559871888s" totalImagesPullingTime="21.021588736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:10.521866115 +0000 UTC m=+96.295798537" watchObservedRunningTime="2026-07-27 20:53:10.559871888 +0000 UTC m=+96.333804315"
Jul 27 20:53:10.618953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.618805    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:10.716075 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:10.715994    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:11.300026 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:11.299951    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pxcw2"
Jul 27 20:53:15.486589 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:15.486553    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:15.582650 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:15.582555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:17.547940 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.547385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pzqf" event={"ID":"f0fc842b-d61e-47d4-9630-b9cd0ba04117","Type":"ContainerStarted","Data":"a6c2693a349b405914f98355b00d7c76ad28964eade0eb592d6f7b61950d9373"}
Jul 27 20:53:17.549956 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.549919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"}
Jul 27 20:53:17.553871 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.553841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm" event={"ID":"63a23771-4518-48de-95d5-92b93d00d0a3","Type":"ContainerStarted","Data":"25535242e9a6cdb247881e5485f5cd587ef8925a0d2b286920efc7dba1f602c5"}
Jul 27 20:53:17.556548 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.556514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn" event={"ID":"251ecbce-92e2-486d-8292-8f593eeac49a","Type":"ContainerStarted","Data":"9434bf54b4770ee9e182d9f2b46718fcbc6588c2024eba7271dc10e5e9c3686d"}
Jul 27 20:53:17.557084 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.557058    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:17.563943 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.563911    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn"
Jul 27 20:53:17.626550 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.626476    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-m69rm" podStartSLOduration=8.832811434 podStartE2EDuration="15.626454121s" totalImagesPullingTime="6.793642687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:17.624527991 +0000 UTC m=+103.398460407" watchObservedRunningTime="2026-07-27 20:53:17.626454121 +0000 UTC m=+103.400386544"
Jul 27 20:53:17.641327 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:17.641265    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-prdpn" podStartSLOduration=3.929186509 podStartE2EDuration="10.641245211s" totalImagesPullingTime="6.712058702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:17.641070189 +0000 UTC m=+103.415002608" watchObservedRunningTime="2026-07-27 20:53:17.641245211 +0000 UTC m=+103.415177634"
Jul 27 20:53:18.161640 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.161596    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:18.262829 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.262790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:18.491542 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.491488    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:18.559700 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.559660    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:18.562164 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.562125    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0fc842b-d61e-47d4-9630-b9cd0ba04117" containerID="a6c2693a349b405914f98355b00d7c76ad28964eade0eb592d6f7b61950d9373" exitCode=0
Jul 27 20:53:18.562313 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.562202    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pzqf" event={"ID":"f0fc842b-d61e-47d4-9630-b9cd0ba04117","Type":"ContainerDied","Data":"a6c2693a349b405914f98355b00d7c76ad28964eade0eb592d6f7b61950d9373"}
Jul 27 20:53:18.563775 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.563749    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b" exitCode=0
Jul 27 20:53:18.563887 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.563844    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"}
Jul 27 20:53:18.566114 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.566090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xpr8" event={"ID":"d37b317e-a9ec-466f-ad1f-4c43d9ccff51","Type":"ContainerStarted","Data":"2928658d1ec73a350bb89a15496e13d598efd59b4594d0e6dc791d56d8f60b3d"}
Jul 27 20:53:18.682342 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:18.682260    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7xpr8" podStartSLOduration=33.012508562 podStartE2EDuration="59.68224519s" totalImagesPullingTime="26.669736628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:52:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:18.68166608 +0000 UTC m=+104.455598503" watchObservedRunningTime="2026-07-27 20:53:18.68224519 +0000 UTC m=+104.456177601"
Jul 27 20:53:19.574989 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:19.574030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pzqf" event={"ID":"f0fc842b-d61e-47d4-9630-b9cd0ba04117","Type":"ContainerStarted","Data":"6aa89cf5d43b6bdffa7b847338e483b97f8680c68346e6e8e19a7b5138d704ae"}
Jul 27 20:53:19.574989 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:19.574130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pzqf" event={"ID":"f0fc842b-d61e-47d4-9630-b9cd0ba04117","Type":"ContainerStarted","Data":"d8d41845cb677e0d17ab64047c7b553cffeb1832975ee0dffd4aa364a3967b64"}
Jul 27 20:53:19.577301 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:19.577262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcb746989-x9mz2" event={"ID":"090cce42-479c-457c-90fc-b7b7b771e0a3","Type":"ContainerStarted","Data":"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"}
Jul 27 20:53:19.612364 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:19.612303    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2pzqf" podStartSLOduration=10.357648577 podStartE2EDuration="17.6122846s" totalImagesPullingTime="7.123503537s" totalInitContainerRuntime="131.132486ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:19.61023506 +0000 UTC m=+105.384167471" watchObservedRunningTime="2026-07-27 20:53:19.6122846 +0000 UTC m=+105.386217021"
Jul 27 20:53:19.650100 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:19.650035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dcb746989-x9mz2" podStartSLOduration=3.759038835 podStartE2EDuration="12.650010445s" totalImagesPullingTime="8.89097161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:19.646569617 +0000 UTC m=+105.420502038" watchObservedRunningTime="2026-07-27 20:53:19.650010445 +0000 UTC m=+105.423942866"
Jul 27 20:53:20.762419 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:20.762375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:20.860837 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:20.860790    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:22.593807 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:22.593774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"}
Jul 27 20:53:22.594199 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:22.593840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"}
Jul 27 20:53:25.339953 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.339869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:25.447409 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.447366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:25.531439 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.531404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:25.612111 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.612025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"}
Jul 27 20:53:25.612111 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.612086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"}
Jul 27 20:53:25.612111 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.612114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"}
Jul 27 20:53:25.612422 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.612138    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerStarted","Data":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"}
Jul 27 20:53:25.651831 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:25.651769    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.983957803 podStartE2EDuration="17.651745867s" totalImagesPullingTime="12.640022942s" totalInitContainerRuntime="27.765122ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:25.649863578 +0000 UTC m=+111.423796001" watchObservedRunningTime="2026-07-27 20:53:25.651745867 +0000 UTC m=+111.425678290"
Jul 27 20:53:27.638708 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:27.638673    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:27.639260 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:27.638745    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:27.643354 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:27.643325    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:28.629510 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:28.629479    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:53:28.799472 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:28.799434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:28.899088 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:28.899000    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:28.963569 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:28.963530    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:29.202048 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:29.201967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:29.630485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:29.630448    2580 generic.go:356] "Generic (PLEG): container finished" podID="68d27cc1-738e-4060-a482-70dac6ca1c6a" containerID="c00afe7a2d94c479a396c3909546b87d34a44652143d290e7a37219623c41bd0" exitCode=0
Jul 27 20:53:29.630772 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:29.630522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m" event={"ID":"68d27cc1-738e-4060-a482-70dac6ca1c6a","Type":"ContainerDied","Data":"c00afe7a2d94c479a396c3909546b87d34a44652143d290e7a37219623c41bd0"}
Jul 27 20:53:29.630974 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:29.630956    2580 scope.go:117] "RemoveContainer" containerID="c00afe7a2d94c479a396c3909546b87d34a44652143d290e7a37219623c41bd0"
Jul 27 20:53:30.637144 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:30.637106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vg56m" event={"ID":"68d27cc1-738e-4060-a482-70dac6ca1c6a","Type":"ContainerStarted","Data":"8063eda933261eff4ec508f8d2d66180b17bde92990888ef8afe066d677bea0a"}
Jul 27 20:53:33.486194 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:33.486145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:33.585036 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:33.584997    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:36.603175 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:36.603140    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:36.672975 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:36.672940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:48.488714 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:48.488674    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:48.551936 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:53:48.551893    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:01.669545 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:01.669505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:09.202106 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:09.202061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:09.219231 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:09.219180    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:09.814058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:09.814029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:21.861435 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:21.861377    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:22.062741 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:22.062697    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:27.300566 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300472    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:27.301018 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300860    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="prometheus" containerID="cri-o://d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d" gracePeriod=600
Jul 27 20:54:27.301018 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300919    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="thanos-sidecar" containerID="cri-o://c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd" gracePeriod=600
Jul 27 20:54:27.301018 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300919    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-thanos" containerID="cri-o://23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052" gracePeriod=600
Jul 27 20:54:27.301018 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300962    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="config-reloader" containerID="cri-o://156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc" gracePeriod=600
Jul 27 20:54:27.301237 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300919    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-web" containerID="cri-o://7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5" gracePeriod=600
Jul 27 20:54:27.301237 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.300916    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy" containerID="cri-o://c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f" gracePeriod=600
Jul 27 20:54:27.554146 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.554062    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:27.623041 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623265 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623049    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623265 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623265 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623087    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623265 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623524 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.623621 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623679 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623639    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623679 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623663    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623778 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623778 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623778 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623741    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623778 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623772    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623970 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623805    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623970 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623835    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.623970 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623888    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624123 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.623941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624176 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.624119    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624255 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.624174    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624255 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.624201    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624359 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.624257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q466q\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q\") pod \"aac66374-5853-4ba9-9f81-247cbb0a3417\" (UID: \"aac66374-5853-4ba9-9f81-247cbb0a3417\") "
Jul 27 20:54:27.624833 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.624560    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.626605 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.626568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.626713 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.626642    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:27.627771 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.627509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q" (OuterVolumeSpecName: "kube-api-access-q466q") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "kube-api-access-q466q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:54:27.627875 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.627806    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:27.628239 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.628166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.628921 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.628689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.628921 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.628704    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.628921 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.628782    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.629944 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.629409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config" (OuterVolumeSpecName: "config") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.629944 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.629721    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.630251 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630179    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.630336 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630271    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.630398 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630338    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:54:27.630668 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.630895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630801    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.630895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630818    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.630895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.630871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out" (OuterVolumeSpecName: "config-out") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:54:27.632205 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.632180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.646158 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.646132    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config" (OuterVolumeSpecName: "web-config") pod "aac66374-5853-4ba9-9f81-247cbb0a3417" (UID: "aac66374-5853-4ba9-9f81-247cbb0a3417"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:54:27.725741 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725707    2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-metrics-client-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725741 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725735    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725741 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725745    2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725755    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725765    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-config-out\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725772    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-tls-assets\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725782    2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725790    2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-grpc-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725798    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725806    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-web-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725814    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725822    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725831    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725839    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725849    2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aac66374-5853-4ba9-9f81-247cbb0a3417-configmap-metrics-client-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725857    2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-kube-rbac-proxy\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725865    2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aac66374-5853-4ba9-9f81-247cbb0a3417-prometheus-k8s-db\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725873    2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aac66374-5853-4ba9-9f81-247cbb0a3417-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.725973 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.725880    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q466q\" (UniqueName: \"kubernetes.io/projected/aac66374-5853-4ba9-9f81-247cbb0a3417-kube-api-access-q466q\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875482    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875510    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875517    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875522    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875527    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875532    2580 generic.go:356] "Generic (PLEG): container finished" podID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d" exitCode=0
Jul 27 20:54:27.875579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875557    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875628    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875564    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875724    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"}
Jul 27 20:54:27.876031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.875760    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aac66374-5853-4ba9-9f81-247cbb0a3417","Type":"ContainerDied","Data":"2f6e602f2ecc18ea8f7f1588f32e32a7dd9192b6fa11f01aaa67b60c2ae9584c"}
Jul 27 20:54:27.885135 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.885111    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.892872 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.892853    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.904096 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.903999    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.904570 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.904500    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:27.906038 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.906020    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:27.912125 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.912105    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.920354 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.920332    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.930555 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.930534    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.938691 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938655    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:27.938954 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938942    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="init-config-reloader"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938957    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="init-config-reloader"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938964    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-thanos"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938971    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-thanos"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938987    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938994    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="thanos-sidecar"
Jul 27 20:54:27.938996 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.938999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="thanos-sidecar"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939007    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="config-reloader"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939012    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="config-reloader"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="prometheus"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939023    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="prometheus"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939034    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-web"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-web"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939100    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="prometheus"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939109    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="config-reloader"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939115    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-thanos"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939120    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939128    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="thanos-sidecar"
Jul 27 20:54:27.939200 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939133    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" containerName="kube-rbac-proxy-web"
Jul 27 20:54:27.939895 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.939876    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.940313 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.940283    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.940402 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.940320    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.940402 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.940355    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.940684 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.940666    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.940734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.940689    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.940734 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.940708    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.940998 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.940979    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.941069 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941002    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.941069 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941018    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.941305 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.941284    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.941401 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941308    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.941401 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941330    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.941587 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.941569    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.941651 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941592    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.941651 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941606    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.941856 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.941840    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.941896 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941858    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.941896 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.941873    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.942127 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:54:27.942086    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.942127 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942107    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.942127 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942122    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.942466 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942434    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.942574 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942468    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.942830 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.942830 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.942822    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.943148 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.943127    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.944370 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.944248    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.944370 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.943456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:27.945296 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.945265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.945296 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.945298    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.945711 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.945685    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.945711 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.945710    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.946019 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.945995    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.946097 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946020    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.946414 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946322    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.946414 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946343    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.946680 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946645    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.946680 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946675    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.947001 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.946974    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.947001 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947000    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.947361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947333    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.947361 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947360    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.947643 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947613    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.947643 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947632    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.947900 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:54:27.947959 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947915    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.947959 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.947938    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.948293 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948273    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:54:27.948388 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948303    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:54:27.948388 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948359    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:54:27.948388 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948354    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.948388 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948386    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.948600 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948471    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:54:27.948653 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:54:27.948701 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948675    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82gkd50gr60i6\""
Jul 27 20:54:27.948753 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.948742    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949252    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949269    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-j724c\""
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949325    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949799    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:54:27.950053 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.949993    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:54:27.950468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.950437    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:54:27.950958 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.950838    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.950958 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.950871    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.955458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.951357    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.955458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.951383    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.955458 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.951392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:54:27.956063 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.955849    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.956063 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.955886    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.956207 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.956183    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.956288 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.956206    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.956906 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.956831    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.956906 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.956871    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.957303 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957271    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.957303 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957304    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.957663 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957626    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.957766 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957666    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.957766 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957708    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:54:27.957987 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957958    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.957987 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.957986    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.958342 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958315    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.958433 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958343    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.958767 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958596    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.958767 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958617    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.958952 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958926    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.959016 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.958954    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.959274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959250    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.959367 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959274    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.959551 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959529    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.959607 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959550    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.959796 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959770    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.959858 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959798    2580 scope.go:117] "RemoveContainer" containerID="23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"
Jul 27 20:54:27.959858 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.959809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:54:27.960033 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960012    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052"} err="failed to get container status \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": rpc error: code = NotFound desc = could not find container \"23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052\": container with ID starting with 23854790b07ce1b661a0f04adb12b76b57606797a0e53e33ce2077d5eb11e052 not found: ID does not exist"
Jul 27 20:54:27.960033 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960029    2580 scope.go:117] "RemoveContainer" containerID="c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"
Jul 27 20:54:27.960257 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960235    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f"} err="failed to get container status \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": rpc error: code = NotFound desc = could not find container \"c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f\": container with ID starting with c2decb2d96ace474d94857eae8c57aba95323ace8d56310d30a2fd6001f7ca6f not found: ID does not exist"
Jul 27 20:54:27.960308 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960261    2580 scope.go:117] "RemoveContainer" containerID="7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"
Jul 27 20:54:27.960505 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960481    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5"} err="failed to get container status \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": rpc error: code = NotFound desc = could not find container \"7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5\": container with ID starting with 7c79ab598ba95e487e645137d5ce847a85a35dfadef860383cce02da7e9d3cc5 not found: ID does not exist"
Jul 27 20:54:27.960560 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960507    2580 scope.go:117] "RemoveContainer" containerID="c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"
Jul 27 20:54:27.960793 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960773    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd"} err="failed to get container status \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": rpc error: code = NotFound desc = could not find container \"c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd\": container with ID starting with c93dd7b769ca7bfad1fa841d6de16e8f8003c45179af489586c8f0bc923b09fd not found: ID does not exist"
Jul 27 20:54:27.960862 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.960795    2580 scope.go:117] "RemoveContainer" containerID="156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"
Jul 27 20:54:27.961138 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.961119    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc"} err="failed to get container status \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": rpc error: code = NotFound desc = could not find container \"156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc\": container with ID starting with 156f29f59eab173029bfb86cd9613e8cf0ef1e283396f087ed26347009708ffc not found: ID does not exist"
Jul 27 20:54:27.961138 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.961138    2580 scope.go:117] "RemoveContainer" containerID="d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"
Jul 27 20:54:27.961490 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.961467    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d"} err="failed to get container status \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": rpc error: code = NotFound desc = could not find container \"d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d\": container with ID starting with d49dca53e784c7bac1dabc8582aa27bf797b221d56d7a047326318b606f64f0d not found: ID does not exist"
Jul 27 20:54:27.961611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.961587    2580 scope.go:117] "RemoveContainer" containerID="f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"
Jul 27 20:54:27.963792 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.963755    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b"} err="failed to get container status \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": rpc error: code = NotFound desc = could not find container \"f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b\": container with ID starting with f2f6e51e483d7b4908142feb5ed56fd0d743d20398018de87a95374a3c6c514b not found: ID does not exist"
Jul 27 20:54:27.971035 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:27.971008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:28.028207 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028207 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028263    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgvl8\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-kube-api-access-qgvl8\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.028826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.029134 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.028847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129467 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129467 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129467 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129687 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129687 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129783 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129740    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129831 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129886 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129938 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.129938 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.129975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130034 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgvl8\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-kube-api-access-qgvl8\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130178    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130317 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.130916 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132524 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.131589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132524 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.132230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132524 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.132476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb56d0eb-a001-4385-ab59-ea3d04b27837-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132726 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.132577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132781 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.132738    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.132781 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.132765    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.133468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.133059    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.133468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.133406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.133468 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.130088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.133666 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.133586    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.133864 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.133839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.134432 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.134413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb56d0eb-a001-4385-ab59-ea3d04b27837-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.135233 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.135191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.135314 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.135301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.135579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.135563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.135616 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.135578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.135696 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.135680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb56d0eb-a001-4385-ab59-ea3d04b27837-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.142052 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.142035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgvl8\" (UniqueName: \"kubernetes.io/projected/fb56d0eb-a001-4385-ab59-ea3d04b27837-kube-api-access-qgvl8\") pod \"prometheus-k8s-0\" (UID: \"fb56d0eb-a001-4385-ab59-ea3d04b27837\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.262592 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.262559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:28.407830 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.407757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:54:28.410624 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:54:28.410582    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb56d0eb_a001_4385_ab59_ea3d04b27837.slice/crio-80d108c989edc6f7c28a6ae77e782adc9d53888556c6c3a0c74b514c7c8175e7 WatchSource:0}: Error finding container 80d108c989edc6f7c28a6ae77e782adc9d53888556c6c3a0c74b514c7c8175e7: Status 404 returned error can't find the container with id 80d108c989edc6f7c28a6ae77e782adc9d53888556c6c3a0c74b514c7c8175e7
Jul 27 20:54:28.804691 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.804661    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac66374-5853-4ba9-9f81-247cbb0a3417" path="/var/lib/kubelet/pods/aac66374-5853-4ba9-9f81-247cbb0a3417/volumes"
Jul 27 20:54:28.881181 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.881147    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb56d0eb-a001-4385-ab59-ea3d04b27837" containerID="e88f8cadcb04872e268cc4b5398a6dd1689c02c69f26ee3cae0b8cb787bfc644" exitCode=0
Jul 27 20:54:28.881391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.881264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerDied","Data":"e88f8cadcb04872e268cc4b5398a6dd1689c02c69f26ee3cae0b8cb787bfc644"}
Jul 27 20:54:28.881391 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:28.881329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"80d108c989edc6f7c28a6ae77e782adc9d53888556c6c3a0c74b514c7c8175e7"}
Jul 27 20:54:29.894896 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.894859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"8bc81944041ce046eeb32e847ec48a09023578e0ce2a541419ee5d08fa1be952"}
Jul 27 20:54:29.895274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.894967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"e8d44db3a7dae56d4ed8d2f4ddeb1943d577bd3e50136fe0278e085be6f599bb"}
Jul 27 20:54:29.895274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.895017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"896623b7ac65af2822efcbed51a3f0b2ee02fdc78f78f56f982865367a329b89"}
Jul 27 20:54:29.895274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.895054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"006747a317f0781ffc2c1f5768a76f9286173ba552826a946a133198e041446f"}
Jul 27 20:54:29.895274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.895097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"6630b003dc8075be9cd0085e08ece0d34e132dc9cce1834611f9f6729230d32c"}
Jul 27 20:54:29.895274 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.895131    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb56d0eb-a001-4385-ab59-ea3d04b27837","Type":"ContainerStarted","Data":"d1c8ad15bee6b8dc3d5bbbd2a72030554c7b1f5d993c4d1ad46fceb20922c5ae"}
Jul 27 20:54:29.930161 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:29.930106    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.912849876 podStartE2EDuration="2.930086885s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.237009ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:54:29.927554635 +0000 UTC m=+175.701487058" watchObservedRunningTime="2026-07-27 20:54:29.930086885 +0000 UTC m=+175.704019310"
Jul 27 20:54:30.929305 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:30.929275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:31.029249 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:31.029191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:31.129017 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:31.128978    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:31.640500 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:31.640463    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:31.705933 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:31.705899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:33.263579 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:33.263540    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:54:41.722854 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:41.722815    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:42.040405 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:42.040315    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:54:42.156790 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:42.156760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:42.256745 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:54:42.256710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:07.061368 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.061329    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dcb746989-x9mz2" podUID="090cce42-479c-457c-90fc-b7b7b771e0a3" containerName="console" containerID="cri-o://16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59" gracePeriod=15
Jul 27 20:55:07.304664 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.304637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:55:07.364323 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364323 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364255    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364323 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzr9m\" (UniqueName: \"kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364376    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364465    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config\") pod \"090cce42-479c-457c-90fc-b7b7b771e0a3\" (UID: \"090cce42-479c-457c-90fc-b7b7b771e0a3\") "
Jul 27 20:55:07.364798 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca" (OuterVolumeSpecName: "service-ca") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:07.364798 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.364731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:07.365051 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.365025    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:07.365398 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.365369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config" (OuterVolumeSpecName: "console-config") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:55:07.366848 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.366824    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:55:07.366960 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.366872    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m" (OuterVolumeSpecName: "kube-api-access-lzr9m") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "kube-api-access-lzr9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:55:07.367020 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.366950    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "090cce42-479c-457c-90fc-b7b7b771e0a3" (UID: "090cce42-479c-457c-90fc-b7b7b771e0a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.465900    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzr9m\" (UniqueName: \"kubernetes.io/projected/090cce42-479c-457c-90fc-b7b7b771e0a3-kube-api-access-lzr9m\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.465956    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-console-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.465970    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-trusted-ca-bundle\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.465982    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-oauth-config\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.465994    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.466005    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/090cce42-479c-457c-90fc-b7b7b771e0a3-console-serving-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:07.466058 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:07.466016    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/090cce42-479c-457c-90fc-b7b7b771e0a3-oauth-serving-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 20:55:08.048830 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.048794    2580 generic.go:356] "Generic (PLEG): container finished" podID="090cce42-479c-457c-90fc-b7b7b771e0a3" containerID="16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59" exitCode=0
Jul 27 20:55:08.048986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.048843    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcb746989-x9mz2"
Jul 27 20:55:08.048986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.048877    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcb746989-x9mz2" event={"ID":"090cce42-479c-457c-90fc-b7b7b771e0a3","Type":"ContainerDied","Data":"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"}
Jul 27 20:55:08.048986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.048925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcb746989-x9mz2" event={"ID":"090cce42-479c-457c-90fc-b7b7b771e0a3","Type":"ContainerDied","Data":"79d90ca4471ccf2343aeb913d108a2491ab326dcf898261aaf37e15ea2a610b4"}
Jul 27 20:55:08.048986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.048947    2580 scope.go:117] "RemoveContainer" containerID="16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"
Jul 27 20:55:08.057522 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.057495    2580 scope.go:117] "RemoveContainer" containerID="16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"
Jul 27 20:55:08.057768 ip-10-0-131-89 kubenswrapper[2580]: E0727 20:55:08.057751    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59\": container with ID starting with 16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59 not found: ID does not exist" containerID="16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"
Jul 27 20:55:08.057819 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.057776    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59"} err="failed to get container status \"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59\": rpc error: code = NotFound desc = could not find container \"16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59\": container with ID starting with 16a33e44870a670318b1be375fc55029e0e35ff94533a7d7942ab63d93e72a59 not found: ID does not exist"
Jul 27 20:55:08.074882 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.074849    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:55:08.078855 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.078829    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dcb746989-x9mz2"]
Jul 27 20:55:08.803740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:08.803703    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="090cce42-479c-457c-90fc-b7b7b771e0a3" path="/var/lib/kubelet/pods/090cce42-479c-457c-90fc-b7b7b771e0a3/volumes"
Jul 27 20:55:10.798207 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:10.798163    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:10.874412 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:10.874375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:28.263007 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:28.262967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:55:28.279561 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:28.279530    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:55:29.150907 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:29.150877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:55:37.200873 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:37.200832    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:37.287485 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:37.287414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:49.120743 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:49.120704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:49.220833 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:49.220792    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:49.790899 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:49.790858    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:49.879182 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:55:49.879147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:01.258969 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:01.258874    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:01.364642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:01.364600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:03.627112 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:03.627074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:03.730005 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:03.729953    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:04.046479 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:04.046445    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:04.661141 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:04.661104    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:34.616256 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:34.616199    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:56:34.616789 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:34.616763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 20:56:34.621079 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:34.621055    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:56:35.913735 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:35.913693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:37.273365 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:56:37.273324    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:01.058535 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:01.058494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:01.418638 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:01.418546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:05.440672 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:05.440638    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:05.543194 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:05.543155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:07.050623 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:07.050581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:07.226323 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:07.226282    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:07.725106 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:07.725034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:07.854452 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:07.854417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:10.093225 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:10.093169    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:10.322943 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:10.322901    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:12.200356 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:12.200319    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:12.271968 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:12.271934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:17.683256 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:17.683193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:18.518085 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:18.518049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:18.647332 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:18.647299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:19.263004 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:19.262929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:19.343420 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:19.343387    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:23.383761 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:23.383723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:23.490020 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:23.489986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:25.094277 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:25.094188    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:40.228471 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:40.228428    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:40.326615 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:40.326539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.019261 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.019196    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.440096 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.440004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.561929 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.561887    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.584373 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.584339    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"]
Jul 27 20:57:41.584758 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.584743    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="090cce42-479c-457c-90fc-b7b7b771e0a3" containerName="console"
Jul 27 20:57:41.584802 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.584761    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="090cce42-479c-457c-90fc-b7b7b771e0a3" containerName="console"
Jul 27 20:57:41.584836 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.584820    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="090cce42-479c-457c-90fc-b7b7b771e0a3" containerName="console"
Jul 27 20:57:41.587642 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.587624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.590724 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.590700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:57:41.590863 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.590729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sqfzw\""
Jul 27 20:57:41.590863 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.590745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:57:41.590863 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.590773    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:57:41.590863 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.590837    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:57:41.615302 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.615272    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"]
Jul 27 20:57:41.656522 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.656486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.656722 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.656536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lfgp\" (UniqueName: \"kubernetes.io/projected/2d46802e-6d8d-41b1-86af-811c999d9b9e-kube-api-access-5lfgp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.656722 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.656571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.656722 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.656660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.656722 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.656691    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.703283 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.703190    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.757342 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lfgp\" (UniqueName: \"kubernetes.io/projected/2d46802e-6d8d-41b1-86af-811c999d9b9e-kube-api-access-5lfgp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.757463 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.757463 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.757463 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.757463 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.757870 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.757849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.759846 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.759822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d46802e-6d8d-41b1-86af-811c999d9b9e-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.760106 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.760088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.760162 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.760136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d46802e-6d8d-41b1-86af-811c999d9b9e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.765852 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.765829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lfgp\" (UniqueName: \"kubernetes.io/projected/2d46802e-6d8d-41b1-86af-811c999d9b9e-kube-api-access-5lfgp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-dqcxf\" (UID: \"2d46802e-6d8d-41b1-86af-811c999d9b9e\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.804663 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.804626    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:41.898098 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.898059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:41.904162 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:41.904126    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:42.022761 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.022730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:42.054587 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.054558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"]
Jul 27 20:57:42.057885 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:57:42.057856    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d46802e_6d8d_41b1_86af_811c999d9b9e.slice/crio-497ca91f676b25ea63d7c31dc76fba5565229dd92173ddd0b1ef6916ef966883 WatchSource:0}: Error finding container 497ca91f676b25ea63d7c31dc76fba5565229dd92173ddd0b1ef6916ef966883: Status 404 returned error can't find the container with id 497ca91f676b25ea63d7c31dc76fba5565229dd92173ddd0b1ef6916ef966883
Jul 27 20:57:42.059791 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.059775    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:57:42.112572 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.112539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:42.216165 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.216131    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:42.313265 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.313157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:42.661563 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:42.661482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf" event={"ID":"2d46802e-6d8d-41b1-86af-811c999d9b9e","Type":"ContainerStarted","Data":"497ca91f676b25ea63d7c31dc76fba5565229dd92173ddd0b1ef6916ef966883"}
Jul 27 20:57:45.208794 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:45.208761    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:45.389715 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:45.389614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:45.489599 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:45.489557    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:45.678854 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:45.678758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf" event={"ID":"2d46802e-6d8d-41b1-86af-811c999d9b9e","Type":"ContainerStarted","Data":"c4b8e52a73f68387109b7aebe0106615dcd23e27c0d3dc0cd7d7ad90db16e252"}
Jul 27 20:57:46.685065 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:46.685028    2580 generic.go:356] "Generic (PLEG): container finished" podID="2d46802e-6d8d-41b1-86af-811c999d9b9e" containerID="c4b8e52a73f68387109b7aebe0106615dcd23e27c0d3dc0cd7d7ad90db16e252" exitCode=0
Jul 27 20:57:46.685462 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:46.685085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf" event={"ID":"2d46802e-6d8d-41b1-86af-811c999d9b9e","Type":"ContainerDied","Data":"c4b8e52a73f68387109b7aebe0106615dcd23e27c0d3dc0cd7d7ad90db16e252"}
Jul 27 20:57:47.691937 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:47.691901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf" event={"ID":"2d46802e-6d8d-41b1-86af-811c999d9b9e","Type":"ContainerStarted","Data":"1c07fce8e6f6dd760c5de5524829108b00e5856127c07d94264d3328643fefab"}
Jul 27 20:57:47.692365 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:47.692026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:57:47.715890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:47.715831    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf" podStartSLOduration=2.3878571109999998 podStartE2EDuration="6.715814169s" totalImagesPullingTime="3.240943848s" totalInitContainerRuntime="1.08701321s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:47.714299294 +0000 UTC m=+373.488231713" watchObservedRunningTime="2026-07-27 20:57:47.715814169 +0000 UTC m=+373.489746588"
Jul 27 20:57:58.699031 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:57:58.698991    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-dqcxf"
Jul 27 20:58:00.666344 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:00.666312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:00.836892 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:00.836859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:01.781897 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:01.781847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:01.905288 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:01.905205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.015611 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:02.015576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.196714 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:02.196614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.446161 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:02.446128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.601803 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:02.601764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:07.904488 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.904451    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"]
Jul 27 20:58:07.910538 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.910511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:07.914402 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 20:58:07.914542 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914433    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 20:58:07.914542 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914445    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:58:07.914542 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914366    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 20:58:07.914542 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 20:58:07.914757 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.914582    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ql59k\""
Jul 27 20:58:07.916559 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.916536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"]
Jul 27 20:58:07.991641 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.991593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7884l\" (UniqueName: \"kubernetes.io/projected/2697bfa4-8005-4ad7-b86e-cf62fc940703-kube-api-access-7884l\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:07.991814 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.991657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:07.991814 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.991684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-metrics-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:07.991814 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:07.991706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2697bfa4-8005-4ad7-b86e-cf62fc940703-manager-config\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.092343 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.092300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.092343 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.092352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-metrics-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.092610 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.092374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2697bfa4-8005-4ad7-b86e-cf62fc940703-manager-config\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.092610 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.092419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7884l\" (UniqueName: \"kubernetes.io/projected/2697bfa4-8005-4ad7-b86e-cf62fc940703-kube-api-access-7884l\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.093128 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.093106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2697bfa4-8005-4ad7-b86e-cf62fc940703-manager-config\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.095001 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.094981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.095096 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.095046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2697bfa4-8005-4ad7-b86e-cf62fc940703-metrics-cert\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.101884 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.101857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7884l\" (UniqueName: \"kubernetes.io/projected/2697bfa4-8005-4ad7-b86e-cf62fc940703-kube-api-access-7884l\") pod \"lws-controller-manager-7f9595876c-tw8ml\" (UID: \"2697bfa4-8005-4ad7-b86e-cf62fc940703\") " pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.220729 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.220692    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:08.358892 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.358855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"]
Jul 27 20:58:08.362065 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:58:08.362041    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2697bfa4_8005_4ad7_b86e_cf62fc940703.slice/crio-60ae48d83814894db2dc5f1bbe887f979fe8b3b693143ecc46dbf662cc74e56a WatchSource:0}: Error finding container 60ae48d83814894db2dc5f1bbe887f979fe8b3b693143ecc46dbf662cc74e56a: Status 404 returned error can't find the container with id 60ae48d83814894db2dc5f1bbe887f979fe8b3b693143ecc46dbf662cc74e56a
Jul 27 20:58:08.785890 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:08.785855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml" event={"ID":"2697bfa4-8005-4ad7-b86e-cf62fc940703","Type":"ContainerStarted","Data":"60ae48d83814894db2dc5f1bbe887f979fe8b3b693143ecc46dbf662cc74e56a"}
Jul 27 20:58:11.791960 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:11.791921    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:11.799412 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:11.799389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml" event={"ID":"2697bfa4-8005-4ad7-b86e-cf62fc940703","Type":"ContainerStarted","Data":"426cf5c8676142ced1d5329f60096ea6ef5b1245f41fc459550686e9bab57939"}
Jul 27 20:58:11.799545 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:11.799520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:58:12.083266 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.083157    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:12.152044 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.151981    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml" podStartSLOduration=2.732257252 podStartE2EDuration="5.151959907s" totalImagesPullingTime="2.419702655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:11.88360138 +0000 UTC m=+397.657533799" watchObservedRunningTime="2026-07-27 20:58:12.151959907 +0000 UTC m=+397.925892327"
Jul 27 20:58:12.152608 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.152582    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-n955m"]
Jul 27 20:58:12.156462 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.156441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.163854 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.163828    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:58:12.164039 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.164025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fnjkr\""
Jul 27 20:58:12.164193 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.164175    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:58:12.187409 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.187376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-n955m"]
Jul 27 20:58:12.225701 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.225669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tls-certs\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.225701 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.225704    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzf4\" (UniqueName: \"kubernetes.io/projected/03ae1502-2e08-4c91-8ce1-4db469bdbaed-kube-api-access-6pzf4\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.225914 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.225785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tmp\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.286600 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.286545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:12.326589 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.326553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tmp\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.326779 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.326644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tls-certs\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.326779 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.326672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzf4\" (UniqueName: \"kubernetes.io/projected/03ae1502-2e08-4c91-8ce1-4db469bdbaed-kube-api-access-6pzf4\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.328936 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.328903    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tmp\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.329101 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.329081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ae1502-2e08-4c91-8ce1-4db469bdbaed-tls-certs\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.336832 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.336779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzf4\" (UniqueName: \"kubernetes.io/projected/03ae1502-2e08-4c91-8ce1-4db469bdbaed-kube-api-access-6pzf4\") pod \"kube-auth-proxy-6598f7b856-n955m\" (UID: \"03ae1502-2e08-4c91-8ce1-4db469bdbaed\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.432009 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.431968    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:12.465506 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.465470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m"
Jul 27 20:58:12.662811 ip-10-0-131-89 kubenswrapper[2580]: W0727 20:58:12.662783    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ae1502_2e08_4c91_8ce1_4db469bdbaed.slice/crio-3b69e695b9b960d7ee2f14a5d3db9871a213d354fa09f7fb18072ea88a920e95 WatchSource:0}: Error finding container 3b69e695b9b960d7ee2f14a5d3db9871a213d354fa09f7fb18072ea88a920e95: Status 404 returned error can't find the container with id 3b69e695b9b960d7ee2f14a5d3db9871a213d354fa09f7fb18072ea88a920e95
Jul 27 20:58:12.685986 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.685950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-n955m"]
Jul 27 20:58:12.804866 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:12.804828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m" event={"ID":"03ae1502-2e08-4c91-8ce1-4db469bdbaed","Type":"ContainerStarted","Data":"3b69e695b9b960d7ee2f14a5d3db9871a213d354fa09f7fb18072ea88a920e95"}
Jul 27 20:58:13.416436 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:13.416394    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:13.583717 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:13.583680    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:15.010726 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:15.010684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:15.120743 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:15.120704    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:15.820740 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:15.820700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m" event={"ID":"03ae1502-2e08-4c91-8ce1-4db469bdbaed","Type":"ContainerStarted","Data":"3812ce35fa8b4a4bd4f31e8378b13bd0146f6cfe75d3617204a25bd07d7d69fd"}
Jul 27 20:58:15.856443 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:15.856389    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6598f7b856-n955m" podStartSLOduration=1.434111906 podStartE2EDuration="3.856372873s" totalImagesPullingTime="2.422260967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:15.854302299 +0000 UTC m=+401.628234719" watchObservedRunningTime="2026-07-27 20:58:15.856372873 +0000 UTC m=+401.630305347"
Jul 27 20:58:16.891887 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:16.891848    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.233972 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:20.233934    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.337712 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:20.337665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.540610 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:20.540522    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.888077 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:20.887983    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:21.067665 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:21.067624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:22.807866 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:58:22.807835    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f9595876c-tw8ml"
Jul 27 20:59:02.186701 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:02.186607    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:02.302646 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:02.302608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:02.507573 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:02.507535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:03.121086 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:03.121043    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:03.324313 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:03.324272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:20.259992 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:20.259951    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:22.097755 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:22.097718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:22.202400 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:22.202357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:31.220320 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:31.220284    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:31.824312 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:31.824272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:32.883334 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:32.883299    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:32.985262 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:32.985199    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:36.437132 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:36.437090    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:36.538650 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:36.538617    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:36.638747 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:36.638710    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:38.838027 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:38.837985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:39.070762 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:39.070727    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:41.270060 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:41.270022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:41.424576 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:41.424537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:44.091541 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:44.091505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:46.227043 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:46.227004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:46.328710 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:46.328676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:46.663912 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:46.663807    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:46.770551 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:46.770517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:47.088998 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:47.088956    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:47.179653 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:47.179615    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:50.246245 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:50.246179    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:50.344374 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:50.344337    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:54.130316 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:54.130272    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:54.442150 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:54.442052    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:54.542491 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:54.542446    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:54.793747 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:54.793678    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:57.819826 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:57.819748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:57.920506 ip-10-0-131-89 kubenswrapper[2580]: I0727 20:59:57.920467    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:05.882061 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:05.882022    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:05.985121 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:05.985080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:06.491055 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:06.491010    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:06.595782 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:06.595741    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:08.659874 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:08.659829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:08.763902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:08.763869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:14.064234 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:14.064176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:14.166967 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:14.166864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:17.461369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:17.461328    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:17.562885 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:17.562825    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:36.506607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:36.506565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:36.973713 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:36.973682    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:37.034814 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:37.034781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:37.165832 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:37.165800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:50.343951 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.343918    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:00:50.347372 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.347353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.348845 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.348816    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:50.349990 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.349973    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 21:00:50.349990 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.349985    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 21:00:50.350987 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.350967    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 21:00:50.350987 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.350974    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lq9jt\""
Jul 27 21:00:50.359070 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.359048    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:00:50.385566 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.385534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.385566 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.385566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrwj\" (UniqueName: \"kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.437597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.437566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:00:50.453824 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.453791    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:50.486983 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.486958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.487132 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.486986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrwj\" (UniqueName: \"kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.487654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.487631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.495308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.495287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrwj\" (UniqueName: \"kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj\") pod \"limitador-limitador-64c94fd64f-992zs\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.554969 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.554931    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:50.658129 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.658043    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:50.795343 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:50.795320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:00:50.799236 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:00:50.799183    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c05e29_6aac_4172_b8f9_dcd62419802e.slice/crio-084e3f4989d06c0281d01eeafcbd2b022ae063c4c76c7e0cd44e31f542d9386e WatchSource:0}: Error finding container 084e3f4989d06c0281d01eeafcbd2b022ae063c4c76c7e0cd44e31f542d9386e: Status 404 returned error can't find the container with id 084e3f4989d06c0281d01eeafcbd2b022ae063c4c76c7e0cd44e31f542d9386e
Jul 27 21:00:51.495268 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.495236    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" event={"ID":"b9c05e29-6aac-4172-b8f9-dcd62419802e","Type":"ContainerStarted","Data":"084e3f4989d06c0281d01eeafcbd2b022ae063c4c76c7e0cd44e31f542d9386e"}
Jul 27 21:00:51.662287 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.662256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:51.729496 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.729462    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:00:51.733855 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.733826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:51.736078 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.735317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:00:51.738621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.738599    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-csp79\""
Jul 27 21:00:51.741598 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.741569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:00:51.799389 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.799321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsg7\" (UniqueName: \"kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7\") pod \"authorino-75d556bd77-q2snb\" (UID: \"711c6be3-1591-4220-b522-1328ca80ab7d\") " pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:00:51.900884 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.900847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsg7\" (UniqueName: \"kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7\") pod \"authorino-75d556bd77-q2snb\" (UID: \"711c6be3-1591-4220-b522-1328ca80ab7d\") " pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:00:51.910748 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:51.910719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsg7\" (UniqueName: \"kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7\") pod \"authorino-75d556bd77-q2snb\" (UID: \"711c6be3-1591-4220-b522-1328ca80ab7d\") " pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:00:52.055283 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:52.055187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:00:52.238412 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:52.238379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:00:52.243485 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:00:52.243442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711c6be3_1591_4220_b522_1328ca80ab7d.slice/crio-d3f989c050d4e1a4776bed35fac0d18f522fd39ba0ed91a429b48de1df28ea75 WatchSource:0}: Error finding container d3f989c050d4e1a4776bed35fac0d18f522fd39ba0ed91a429b48de1df28ea75: Status 404 returned error can't find the container with id d3f989c050d4e1a4776bed35fac0d18f522fd39ba0ed91a429b48de1df28ea75
Jul 27 21:00:52.501810 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:52.501773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2snb" event={"ID":"711c6be3-1591-4220-b522-1328ca80ab7d","Type":"ContainerStarted","Data":"d3f989c050d4e1a4776bed35fac0d18f522fd39ba0ed91a429b48de1df28ea75"}
Jul 27 21:00:54.518626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:54.518583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" event={"ID":"b9c05e29-6aac-4172-b8f9-dcd62419802e","Type":"ContainerStarted","Data":"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"}
Jul 27 21:00:54.519074 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:54.518712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:00:54.536297 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:54.536247    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" podStartSLOduration=1.287706582 podStartE2EDuration="4.536230767s" totalImagesPullingTime="3.248524185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:54.533912567 +0000 UTC m=+560.307844988" watchObservedRunningTime="2026-07-27 21:00:54.536230767 +0000 UTC m=+560.310163183"
Jul 27 21:00:56.530114 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:56.530073    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2snb" event={"ID":"711c6be3-1591-4220-b522-1328ca80ab7d","Type":"ContainerStarted","Data":"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"}
Jul 27 21:00:56.545457 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:00:56.545406    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q2snb" podStartSLOduration=1.8403238910000002 podStartE2EDuration="5.545389948s" totalImagesPullingTime="3.705066057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:56.544120507 +0000 UTC m=+562.318052927" watchObservedRunningTime="2026-07-27 21:00:56.545389948 +0000 UTC m=+562.319322370"
Jul 27 21:01:05.271176 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.271132    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:01:05.271668 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.271373    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" podUID="b9c05e29-6aac-4172-b8f9-dcd62419802e" containerName="limitador" containerID="cri-o://b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc" gracePeriod=30
Jul 27 21:01:05.272015 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.271920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:01:05.812812 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.812786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:01:05.924912 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.924827    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file\") pod \"b9c05e29-6aac-4172-b8f9-dcd62419802e\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") "
Jul 27 21:01:05.924912 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.924902    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqrwj\" (UniqueName: \"kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj\") pod \"b9c05e29-6aac-4172-b8f9-dcd62419802e\" (UID: \"b9c05e29-6aac-4172-b8f9-dcd62419802e\") "
Jul 27 21:01:05.925264 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.925239    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file" (OuterVolumeSpecName: "config-file") pod "b9c05e29-6aac-4172-b8f9-dcd62419802e" (UID: "b9c05e29-6aac-4172-b8f9-dcd62419802e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:01:05.927169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:05.927143    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj" (OuterVolumeSpecName: "kube-api-access-gqrwj") pod "b9c05e29-6aac-4172-b8f9-dcd62419802e" (UID: "b9c05e29-6aac-4172-b8f9-dcd62419802e"). InnerVolumeSpecName "kube-api-access-gqrwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:01:06.025888 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.025861    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrwj\" (UniqueName: \"kubernetes.io/projected/b9c05e29-6aac-4172-b8f9-dcd62419802e-kube-api-access-gqrwj\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:06.025888 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.025883    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b9c05e29-6aac-4172-b8f9-dcd62419802e-config-file\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:06.574737 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.574699    2580 generic.go:356] "Generic (PLEG): container finished" podID="b9c05e29-6aac-4172-b8f9-dcd62419802e" containerID="b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc" exitCode=0
Jul 27 21:01:06.574737 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.574738    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs"
Jul 27 21:01:06.575243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.574782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" event={"ID":"b9c05e29-6aac-4172-b8f9-dcd62419802e","Type":"ContainerDied","Data":"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"}
Jul 27 21:01:06.575243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.574858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-992zs" event={"ID":"b9c05e29-6aac-4172-b8f9-dcd62419802e","Type":"ContainerDied","Data":"084e3f4989d06c0281d01eeafcbd2b022ae063c4c76c7e0cd44e31f542d9386e"}
Jul 27 21:01:06.575243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.574884    2580 scope.go:117] "RemoveContainer" containerID="b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"
Jul 27 21:01:06.583716 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.583700    2580 scope.go:117] "RemoveContainer" containerID="b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"
Jul 27 21:01:06.583933 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:06.583918    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc\": container with ID starting with b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc not found: ID does not exist" containerID="b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"
Jul 27 21:01:06.583977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.583940    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc"} err="failed to get container status \"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc\": rpc error: code = NotFound desc = could not find container \"b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc\": container with ID starting with b5552ed32c5ede2bacdc1a7cc6b590e329dd329a12e8ffe61bf1491e64c0fadc not found: ID does not exist"
Jul 27 21:01:06.595041 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.595017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:01:06.598945 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.598922    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-992zs"]
Jul 27 21:01:06.804982 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:06.804948    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c05e29-6aac-4172-b8f9-dcd62419802e" path="/var/lib/kubelet/pods/b9c05e29-6aac-4172-b8f9-dcd62419802e/volumes"
Jul 27 21:01:09.083443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:09.083409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:09.136738 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:09.136702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:10.310752 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.310717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:10.410980 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.410944    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:10.726988 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.726955    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"]
Jul 27 21:01:10.727299 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.727286    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c05e29-6aac-4172-b8f9-dcd62419802e" containerName="limitador"
Jul 27 21:01:10.727349 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.727300    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c05e29-6aac-4172-b8f9-dcd62419802e" containerName="limitador"
Jul 27 21:01:10.727384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.727364    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c05e29-6aac-4172-b8f9-dcd62419802e" containerName="limitador"
Jul 27 21:01:10.731313 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.731286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.733739 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.733719    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 27 21:01:10.733870 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.733747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j6nf4\""
Jul 27 21:01:10.734617 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.734598    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:01:10.734729 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.734628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:01:10.740890 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.740867    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"]
Jul 27 21:01:10.776082 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.776059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:10.864979 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.864957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15c864a5-74fc-4826-9dc6-1b704acb6e63-data\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.865096 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.864999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfqq\" (UniqueName: \"kubernetes.io/projected/15c864a5-74fc-4826-9dc6-1b704acb6e63-kube-api-access-wmfqq\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.966044 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.966015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15c864a5-74fc-4826-9dc6-1b704acb6e63-data\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.966162 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.966070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfqq\" (UniqueName: \"kubernetes.io/projected/15c864a5-74fc-4826-9dc6-1b704acb6e63-kube-api-access-wmfqq\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.966420 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.966396    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/15c864a5-74fc-4826-9dc6-1b704acb6e63-data\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:10.975642 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:10.975616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfqq\" (UniqueName: \"kubernetes.io/projected/15c864a5-74fc-4826-9dc6-1b704acb6e63-kube-api-access-wmfqq\") pod \"postgres-69fb4594cf-kblm2\" (UID: \"15c864a5-74fc-4826-9dc6-1b704acb6e63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:11.043583 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:11.043519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:11.171854 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:11.171710    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"]
Jul 27 21:01:11.174132 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:01:11.174108    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c864a5_74fc_4826_9dc6_1b704acb6e63.slice/crio-572d4a3f190389a75ec42fc0b50a063c975b3d28c3033dd59f1e328c1c7517bd WatchSource:0}: Error finding container 572d4a3f190389a75ec42fc0b50a063c975b3d28c3033dd59f1e328c1c7517bd: Status 404 returned error can't find the container with id 572d4a3f190389a75ec42fc0b50a063c975b3d28c3033dd59f1e328c1c7517bd
Jul 27 21:01:11.598455 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:11.598416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2" event={"ID":"15c864a5-74fc-4826-9dc6-1b704acb6e63","Type":"ContainerStarted","Data":"572d4a3f190389a75ec42fc0b50a063c975b3d28c3033dd59f1e328c1c7517bd"}
Jul 27 21:01:13.727621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:13.727576    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:13.782542 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:13.782498    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:17.631883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:17.631845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2" event={"ID":"15c864a5-74fc-4826-9dc6-1b704acb6e63","Type":"ContainerStarted","Data":"2b40382bf280870385e0f4cba0e43b03fe6b3ae35ec5e2e5a3ed6bf45e96b3df"}
Jul 27 21:01:17.632310 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:17.632033    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:17.648139 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:17.648075    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2" podStartSLOduration=2.225851558 podStartE2EDuration="7.648058762s" totalImagesPullingTime="5.422207204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:01:17.64618656 +0000 UTC m=+583.420118982" watchObservedRunningTime="2026-07-27 21:01:17.648058762 +0000 UTC m=+583.421991181"
Jul 27 21:01:23.667315 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:23.667277    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kblm2"
Jul 27 21:01:23.991893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:23.991859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:24.219429 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.219397    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rnjqb"]
Jul 27 21:01:24.225959 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.225939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.229225 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.229186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rnjqb"]
Jul 27 21:01:24.383795 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.383720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vw2t\" (UniqueName: \"kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t\") pod \"authorino-5d4b67c446-rnjqb\" (UID: \"44896d4d-c085-4f6e-85db-6d15c028439d\") " pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.392008 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.391984    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:24.464732 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.464702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rnjqb"]
Jul 27 21:01:24.464976 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:24.464958    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-9vw2t], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-rnjqb" podUID="44896d4d-c085-4f6e-85db-6d15c028439d"
Jul 27 21:01:24.484481 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.484451    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vw2t\" (UniqueName: \"kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t\") pod \"authorino-5d4b67c446-rnjqb\" (UID: \"44896d4d-c085-4f6e-85db-6d15c028439d\") " pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.491837 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.491820    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vw2t\" (UniqueName: \"kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t\") pod \"authorino-5d4b67c446-rnjqb\" (UID: \"44896d4d-c085-4f6e-85db-6d15c028439d\") " pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.666005 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.665922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.673477 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.673454    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:24.787155 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.787125    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vw2t\" (UniqueName: \"kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t\") pod \"44896d4d-c085-4f6e-85db-6d15c028439d\" (UID: \"44896d4d-c085-4f6e-85db-6d15c028439d\") "
Jul 27 21:01:24.789293 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.789269    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t" (OuterVolumeSpecName: "kube-api-access-9vw2t") pod "44896d4d-c085-4f6e-85db-6d15c028439d" (UID: "44896d4d-c085-4f6e-85db-6d15c028439d"). InnerVolumeSpecName "kube-api-access-9vw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:01:24.888125 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:24.888080    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vw2t\" (UniqueName: \"kubernetes.io/projected/44896d4d-c085-4f6e-85db-6d15c028439d-kube-api-access-9vw2t\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:25.670559 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:25.670519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rnjqb"
Jul 27 21:01:25.695080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:25.695047    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rnjqb"]
Jul 27 21:01:25.698705 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:25.698674    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rnjqb"]
Jul 27 21:01:26.805180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:26.805143    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44896d4d-c085-4f6e-85db-6d15c028439d" path="/var/lib/kubelet/pods/44896d4d-c085-4f6e-85db-6d15c028439d/volumes"
Jul 27 21:01:27.063872 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.063800    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:01:27.064004 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.063984    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q2snb" podUID="711c6be3-1591-4220-b522-1328ca80ab7d" containerName="authorino" containerID="cri-o://53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39" gracePeriod=30
Jul 27 21:01:27.371651 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.371612    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:01:27.509971 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.509938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djsg7\" (UniqueName: \"kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7\") pod \"711c6be3-1591-4220-b522-1328ca80ab7d\" (UID: \"711c6be3-1591-4220-b522-1328ca80ab7d\") "
Jul 27 21:01:27.512134 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.512112    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7" (OuterVolumeSpecName: "kube-api-access-djsg7") pod "711c6be3-1591-4220-b522-1328ca80ab7d" (UID: "711c6be3-1591-4220-b522-1328ca80ab7d"). InnerVolumeSpecName "kube-api-access-djsg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:01:27.611516 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.611435    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djsg7\" (UniqueName: \"kubernetes.io/projected/711c6be3-1591-4220-b522-1328ca80ab7d-kube-api-access-djsg7\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:27.681366 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.681332    2580 generic.go:356] "Generic (PLEG): container finished" podID="711c6be3-1591-4220-b522-1328ca80ab7d" containerID="53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39" exitCode=0
Jul 27 21:01:27.681538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.681373    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q2snb"
Jul 27 21:01:27.681538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.681409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2snb" event={"ID":"711c6be3-1591-4220-b522-1328ca80ab7d","Type":"ContainerDied","Data":"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"}
Jul 27 21:01:27.681538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.681454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q2snb" event={"ID":"711c6be3-1591-4220-b522-1328ca80ab7d","Type":"ContainerDied","Data":"d3f989c050d4e1a4776bed35fac0d18f522fd39ba0ed91a429b48de1df28ea75"}
Jul 27 21:01:27.681538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.681474    2580 scope.go:117] "RemoveContainer" containerID="53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"
Jul 27 21:01:27.691193 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.691171    2580 scope.go:117] "RemoveContainer" containerID="53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"
Jul 27 21:01:27.691585 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:27.691565    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39\": container with ID starting with 53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39 not found: ID does not exist" containerID="53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"
Jul 27 21:01:27.691647 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.691596    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39"} err="failed to get container status \"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39\": rpc error: code = NotFound desc = could not find container \"53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39\": container with ID starting with 53b468c8607f4ca1a07c7bd851e57630e1d44c4df6c3f2dc70d592fed786cf39 not found: ID does not exist"
Jul 27 21:01:27.705538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.705494    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:01:27.714783 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:27.714746    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q2snb"]
Jul 27 21:01:28.804806 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:28.804771    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711c6be3-1591-4220-b522-1328ca80ab7d" path="/var/lib/kubelet/pods/711c6be3-1591-4220-b522-1328ca80ab7d/volumes"
Jul 27 21:01:31.929308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:31.929275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:31.950232 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:31.950188    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:31.950563 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:31.950549    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711c6be3-1591-4220-b522-1328ca80ab7d" containerName="authorino"
Jul 27 21:01:31.950610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:31.950567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="711c6be3-1591-4220-b522-1328ca80ab7d" containerName="authorino"
Jul 27 21:01:31.950644 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:31.950634    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="711c6be3-1591-4220-b522-1328ca80ab7d" containerName="authorino"
Jul 27 21:01:32.040703 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.040671    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:32.120147 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.120115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:32.120297 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.120239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.123032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.122988    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:01:32.123177 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.123131    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zdvqq\""
Jul 27 21:01:32.123177 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.123166    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:01:32.126429 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.126404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:32.153887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.153859    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:01:32.176929 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.176905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:01:32.177055 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.177000    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.254546 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.254514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.254698 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.254558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5ph\" (UniqueName: \"kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.355368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.355338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.355515 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.355375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qf9\" (UniqueName: \"kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.355515 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.355416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5ph\" (UniqueName: \"kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.355515 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.355475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.355515 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:32.355492    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 27 21:01:32.355715 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:32.355550    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert podName:2392f1ca-8eb4-4cfa-a367-322efad83c9a nodeName:}" failed. No retries permitted until 2026-07-27 21:01:32.855531185 +0000 UTC m=+598.629463587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert") pod "maas-controller-7ccfb6dd45-4f77k" (UID: "2392f1ca-8eb4-4cfa-a367-322efad83c9a") : secret "maas-controller-webhook-cert" not found
Jul 27 21:01:32.365774 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.365745    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5ph\" (UniqueName: \"kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.455319 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.455246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:32.456262 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.456246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qf9\" (UniqueName: \"kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.456344 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.456295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.458653 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.458634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.464248 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.464233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qf9\" (UniqueName: \"kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9\") pod \"maas-controller-55f67d7fff-2p6jb\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") " pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.486722 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.486703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:32.618687 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.618498    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:01:32.621412 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:01:32.621382    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85edd041_2fc1_4a03_a1d1_da900a44151e.slice/crio-b3c585e58b9096df86adef7533e4f2ee512c91a367996c7cc45a316ca734347a WatchSource:0}: Error finding container b3c585e58b9096df86adef7533e4f2ee512c91a367996c7cc45a316ca734347a: Status 404 returned error can't find the container with id b3c585e58b9096df86adef7533e4f2ee512c91a367996c7cc45a316ca734347a
Jul 27 21:01:32.707193 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.707164    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" event={"ID":"85edd041-2fc1-4a03-a1d1-da900a44151e","Type":"ContainerStarted","Data":"b3c585e58b9096df86adef7533e4f2ee512c91a367996c7cc45a316ca734347a"}
Jul 27 21:01:32.859478 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.859439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:32.861941 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:32.861919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") pod \"maas-controller-7ccfb6dd45-4f77k\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") " pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:33.030858 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:33.030773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:33.161725 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:33.161697    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:33.164280 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:01:33.164258    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2392f1ca_8eb4_4cfa_a367_322efad83c9a.slice/crio-cbed520775f72537f04c39e84ecba7a6c140ed4d80c6ab5556f5720b4f9bc1f9 WatchSource:0}: Error finding container cbed520775f72537f04c39e84ecba7a6c140ed4d80c6ab5556f5720b4f9bc1f9: Status 404 returned error can't find the container with id cbed520775f72537f04c39e84ecba7a6c140ed4d80c6ab5556f5720b4f9bc1f9
Jul 27 21:01:33.714023 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:33.713974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" event={"ID":"2392f1ca-8eb4-4cfa-a367-322efad83c9a","Type":"ContainerStarted","Data":"cbed520775f72537f04c39e84ecba7a6c140ed4d80c6ab5556f5720b4f9bc1f9"}
Jul 27 21:01:35.566822 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.566792    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:01:35.567249 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.567233    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:01:35.582974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.582947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:01:35.728146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.728102    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" event={"ID":"85edd041-2fc1-4a03-a1d1-da900a44151e","Type":"ContainerStarted","Data":"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"}
Jul 27 21:01:35.728330 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.728306    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:35.729609 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.729583    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" event={"ID":"2392f1ca-8eb4-4cfa-a367-322efad83c9a","Type":"ContainerStarted","Data":"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"}
Jul 27 21:01:35.729764 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.729750    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:35.766650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.766579    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:35.806260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.806135    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" podStartSLOduration=0.844662244 podStartE2EDuration="3.806118104s" totalImagesPullingTime="2.96145586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:01:35.762664565 +0000 UTC m=+601.536596985" watchObservedRunningTime="2026-07-27 21:01:35.806118104 +0000 UTC m=+601.580050544"
Jul 27 21:01:35.806615 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.806590    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" podStartSLOduration=2.396158343 podStartE2EDuration="4.806583418s" totalImagesPullingTime="2.410425075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:01:35.805621169 +0000 UTC m=+601.579553590" watchObservedRunningTime="2026-07-27 21:01:35.806583418 +0000 UTC m=+601.580515838"
Jul 27 21:01:35.894485 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:35.894453    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:46.736941 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:46.736912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:01:46.738530 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:46.737003    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:46.806355 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:46.806322    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:46.806567 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:46.806537    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" podUID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" containerName="manager" containerID="cri-o://f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab" gracePeriod=30
Jul 27 21:01:47.047120 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.047092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:47.081478 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.081445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") pod \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") "
Jul 27 21:01:47.081666 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.081512    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm5ph\" (UniqueName: \"kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph\") pod \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\" (UID: \"2392f1ca-8eb4-4cfa-a367-322efad83c9a\") "
Jul 27 21:01:47.083922 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.083879    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2392f1ca-8eb4-4cfa-a367-322efad83c9a" (UID: "2392f1ca-8eb4-4cfa-a367-322efad83c9a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:01:47.084036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.083962    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph" (OuterVolumeSpecName: "kube-api-access-dm5ph") pod "2392f1ca-8eb4-4cfa-a367-322efad83c9a" (UID: "2392f1ca-8eb4-4cfa-a367-322efad83c9a"). InnerVolumeSpecName "kube-api-access-dm5ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:01:47.183392 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.183355    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2392f1ca-8eb4-4cfa-a367-322efad83c9a-webhook-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:47.183571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.183416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5ph\" (UniqueName: \"kubernetes.io/projected/2392f1ca-8eb4-4cfa-a367-322efad83c9a-kube-api-access-dm5ph\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:01:47.787300 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.787265    2580 generic.go:356] "Generic (PLEG): container finished" podID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" containerID="f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab" exitCode=0
Jul 27 21:01:47.787689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.787316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k"
Jul 27 21:01:47.787689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.787352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" event={"ID":"2392f1ca-8eb4-4cfa-a367-322efad83c9a","Type":"ContainerDied","Data":"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"}
Jul 27 21:01:47.787689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.787402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7ccfb6dd45-4f77k" event={"ID":"2392f1ca-8eb4-4cfa-a367-322efad83c9a","Type":"ContainerDied","Data":"cbed520775f72537f04c39e84ecba7a6c140ed4d80c6ab5556f5720b4f9bc1f9"}
Jul 27 21:01:47.787689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.787424    2580 scope.go:117] "RemoveContainer" containerID="f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"
Jul 27 21:01:47.797080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.797058    2580 scope.go:117] "RemoveContainer" containerID="f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"
Jul 27 21:01:47.797420 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:01:47.797398    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab\": container with ID starting with f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab not found: ID does not exist" containerID="f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"
Jul 27 21:01:47.797479 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.797431    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab"} err="failed to get container status \"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab\": rpc error: code = NotFound desc = could not find container \"f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab\": container with ID starting with f8d1d38ca976a1bd4c84d83afc1b2e8b089de4cb7704ce2b4a27c80f14ca96ab not found: ID does not exist"
Jul 27 21:01:47.809308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.809280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:47.813199 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:47.813166    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7ccfb6dd45-4f77k"]
Jul 27 21:01:48.804508 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:48.804469    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" path="/var/lib/kubelet/pods/2392f1ca-8eb4-4cfa-a367-322efad83c9a/volumes"
Jul 27 21:01:50.495761 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:50.495724    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:52.317383 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:01:52.317347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:08.541108 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:08.540998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:08.682572 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:08.682534    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:09.113814 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.113778    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:09.267994 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.267956    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"]
Jul 27 21:02:09.268416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.268394    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" containerName="manager"
Jul 27 21:02:09.268416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.268411    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" containerName="manager"
Jul 27 21:02:09.268571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.268472    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2392f1ca-8eb4-4cfa-a367-322efad83c9a" containerName="manager"
Jul 27 21:02:09.273992 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.273966    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.277596 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.277574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jc7bw\""
Jul 27 21:02:09.278033 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.278017    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 27 21:02:09.284105 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.284077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"]
Jul 27 21:02:09.391359 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.391273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfng\" (UniqueName: \"kubernetes.io/projected/357eb02d-52e6-4fe3-874d-d5df4d85b887-kube-api-access-phfng\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.391359 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.391308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357eb02d-52e6-4fe3-874d-d5df4d85b887-openshift-service-ca\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.391553 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.391400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357eb02d-52e6-4fe3-874d-d5df4d85b887-maas-api-tls\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.478850 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.478817    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:09.492891 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.492831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfng\" (UniqueName: \"kubernetes.io/projected/357eb02d-52e6-4fe3-874d-d5df4d85b887-kube-api-access-phfng\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.492891 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.492901    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357eb02d-52e6-4fe3-874d-d5df4d85b887-openshift-service-ca\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.493073 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.492978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357eb02d-52e6-4fe3-874d-d5df4d85b887-maas-api-tls\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.493652 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.493623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/357eb02d-52e6-4fe3-874d-d5df4d85b887-openshift-service-ca\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.495559 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.495531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/357eb02d-52e6-4fe3-874d-d5df4d85b887-maas-api-tls\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.501294 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.501266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfng\" (UniqueName: \"kubernetes.io/projected/357eb02d-52e6-4fe3-874d-d5df4d85b887-kube-api-access-phfng\") pod \"maas-api-5b64468d9c-hqrbc\" (UID: \"357eb02d-52e6-4fe3-874d-d5df4d85b887\") " pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.585498 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.585456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:09.737850 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.737822    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"]
Jul 27 21:02:09.740147 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:02:09.740120    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357eb02d_52e6_4fe3_874d_d5df4d85b887.slice/crio-7d6c0f9a218c2aadeb6ec1bdc53522dc49dae50477adf8e1c50b79c7360dba3b WatchSource:0}: Error finding container 7d6c0f9a218c2aadeb6ec1bdc53522dc49dae50477adf8e1c50b79c7360dba3b: Status 404 returned error can't find the container with id 7d6c0f9a218c2aadeb6ec1bdc53522dc49dae50477adf8e1c50b79c7360dba3b
Jul 27 21:02:09.802277 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.802228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:09.892765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.892731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc" event={"ID":"357eb02d-52e6-4fe3-874d-d5df4d85b887","Type":"ContainerStarted","Data":"7d6c0f9a218c2aadeb6ec1bdc53522dc49dae50477adf8e1c50b79c7360dba3b"}
Jul 27 21:02:09.902134 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.902105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:09.939156 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.939070    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5jb2v"]
Jul 27 21:02:09.949503 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.949466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:09.953545 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.953517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5jb2v"]
Jul 27 21:02:09.955617 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.955590    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 27 21:02:09.955763 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.955591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tthd2\""
Jul 27 21:02:09.990322 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.990286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"]
Jul 27 21:02:09.993511 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:09.993494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.005022 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.004988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"]
Jul 27 21:02:10.098506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.098463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tqr\" (UniqueName: \"kubernetes.io/projected/8deca1b0-401f-47eb-b698-36f931657c2c-kube-api-access-96tqr\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.098506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.098512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7vx\" (UniqueName: \"kubernetes.io/projected/a9ffe201-d02b-4136-893d-656f80bb5e36-kube-api-access-9m7vx\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.098946 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.098651    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ffe201-d02b-4136-893d-656f80bb5e36-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.098946 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.098700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deca1b0-401f-47eb-b698-36f931657c2c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.199444 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.199343    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ffe201-d02b-4136-893d-656f80bb5e36-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.199444 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.199391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deca1b0-401f-47eb-b698-36f931657c2c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.199444 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.199429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tqr\" (UniqueName: \"kubernetes.io/projected/8deca1b0-401f-47eb-b698-36f931657c2c-kube-api-access-96tqr\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.199732 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.199468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7vx\" (UniqueName: \"kubernetes.io/projected/a9ffe201-d02b-4136-893d-656f80bb5e36-kube-api-access-9m7vx\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.200063 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.200041    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9ffe201-d02b-4136-893d-656f80bb5e36-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.200119 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.200079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8deca1b0-401f-47eb-b698-36f931657c2c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.207706 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.207683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tqr\" (UniqueName: \"kubernetes.io/projected/8deca1b0-401f-47eb-b698-36f931657c2c-kube-api-access-96tqr\") pod \"payload-processing-5cc6cd444b-5jb2v\" (UID: \"8deca1b0-401f-47eb-b698-36f931657c2c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.207784 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.207748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7vx\" (UniqueName: \"kubernetes.io/projected/a9ffe201-d02b-4136-893d-656f80bb5e36-kube-api-access-9m7vx\") pod \"payload-pre-processing-687dbb6cf6-pxn4h\" (UID: \"a9ffe201-d02b-4136-893d-656f80bb5e36\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.264528 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.264488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:10.302940 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.302904    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:10.413789 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.413735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5jb2v"]
Jul 27 21:02:10.417565 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:02:10.417530    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deca1b0_401f_47eb_b698_36f931657c2c.slice/crio-b2c705147c8a8c3d2f81e526a629ac3d1d6a314360bfdc2dc184a20d596222b2 WatchSource:0}: Error finding container b2c705147c8a8c3d2f81e526a629ac3d1d6a314360bfdc2dc184a20d596222b2: Status 404 returned error can't find the container with id b2c705147c8a8c3d2f81e526a629ac3d1d6a314360bfdc2dc184a20d596222b2
Jul 27 21:02:10.470148 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.469965    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"]
Jul 27 21:02:10.472860 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:02:10.472833    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ffe201_d02b_4136_893d_656f80bb5e36.slice/crio-8bb6368bb5b46675c73d356680929da6b60500e4773d4398743773c98ba0042e WatchSource:0}: Error finding container 8bb6368bb5b46675c73d356680929da6b60500e4773d4398743773c98ba0042e: Status 404 returned error can't find the container with id 8bb6368bb5b46675c73d356680929da6b60500e4773d4398743773c98ba0042e
Jul 27 21:02:10.898738 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.898640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h" event={"ID":"a9ffe201-d02b-4136-893d-656f80bb5e36","Type":"ContainerStarted","Data":"8bb6368bb5b46675c73d356680929da6b60500e4773d4398743773c98ba0042e"}
Jul 27 21:02:10.899786 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:10.899761    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v" event={"ID":"8deca1b0-401f-47eb-b698-36f931657c2c","Type":"ContainerStarted","Data":"b2c705147c8a8c3d2f81e526a629ac3d1d6a314360bfdc2dc184a20d596222b2"}
Jul 27 21:02:13.919735 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.919703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc" event={"ID":"357eb02d-52e6-4fe3-874d-d5df4d85b887","Type":"ContainerStarted","Data":"4d037c3ee9c97cca4a3d9fc69a55f890037c8e4d0253e54ab9b6d85b213ba039"}
Jul 27 21:02:13.920194 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.919838    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:13.921358 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.921333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h" event={"ID":"a9ffe201-d02b-4136-893d-656f80bb5e36","Type":"ContainerStarted","Data":"c0c1ba2bddc56f39f2a89bd59aea44395cb15fbe4ff999fff34e223ed36ee01c"}
Jul 27 21:02:13.921475 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.921412    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:13.922745 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.922726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v" event={"ID":"8deca1b0-401f-47eb-b698-36f931657c2c","Type":"ContainerStarted","Data":"bb216b3e698b5baed5ca516a1d2f482673f69bc7f9e4e8425218cbaa06e210cb"}
Jul 27 21:02:13.922828 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.922801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:13.936593 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.936551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc" podStartSLOduration=1.648714486 podStartE2EDuration="4.936537011s" totalImagesPullingTime="3.287822525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:13.936079931 +0000 UTC m=+639.710012352" watchObservedRunningTime="2026-07-27 21:02:13.936537011 +0000 UTC m=+639.710469430"
Jul 27 21:02:13.954278 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.954197    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v" podStartSLOduration=2.26855537 podStartE2EDuration="4.954180466s" totalImagesPullingTime="2.685625096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:13.952047107 +0000 UTC m=+639.725979531" watchObservedRunningTime="2026-07-27 21:02:13.954180466 +0000 UTC m=+639.728112885"
Jul 27 21:02:13.968428 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:13.968381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h" podStartSLOduration=2.34297334 podStartE2EDuration="4.968364439s" totalImagesPullingTime="2.625391099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:13.966475724 +0000 UTC m=+639.740408144" watchObservedRunningTime="2026-07-27 21:02:13.968364439 +0000 UTC m=+639.742296858"
Jul 27 21:02:19.933090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:19.933059    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b64468d9c-hqrbc"
Jul 27 21:02:20.990452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:20.990415    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:21.090815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:21.090756    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:24.929634 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:24.929602    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5jb2v"
Jul 27 21:02:24.930031 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:24.929670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pxn4h"
Jul 27 21:02:45.232248 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:45.232187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:56.358894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:56.358859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:59.732108 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.732068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"]
Jul 27 21:02:59.734485 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.734467    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.739974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.739950    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bt4k8\""
Jul 27 21:02:59.740097 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.739967    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 27 21:02:59.740195 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.739981    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 21:02:59.740285 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.740029    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 21:02:59.751999 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.751973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"]
Jul 27 21:02:59.774316 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.774287    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:59.833932 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.833897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.834156 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.833954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83986328-9388-4556-b86f-b971c9d17d98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.834156 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.834014    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.834156 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.834054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.834156 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.834084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.834396 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.834194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxbn\" (UniqueName: \"kubernetes.io/projected/83986328-9388-4556-b86f-b971c9d17d98-kube-api-access-zpxbn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.935712 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.935883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83986328-9388-4556-b86f-b971c9d17d98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.935883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.935883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.935883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.936049 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.935898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxbn\" (UniqueName: \"kubernetes.io/projected/83986328-9388-4556-b86f-b971c9d17d98-kube-api-access-zpxbn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.936175 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.936153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.936263 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.936193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.936327 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.936314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.938251 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.938233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83986328-9388-4556-b86f-b971c9d17d98-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.938514 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.938498    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83986328-9388-4556-b86f-b971c9d17d98-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:02:59.944588 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:02:59.944566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxbn\" (UniqueName: \"kubernetes.io/projected/83986328-9388-4556-b86f-b971c9d17d98-kube-api-access-zpxbn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ql9qx\" (UID: \"83986328-9388-4556-b86f-b971c9d17d98\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:00.044491 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:00.044400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:00.189111 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:00.189079    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"]
Jul 27 21:03:00.192425 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:03:00.192394    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83986328_9388_4556_b86f_b971c9d17d98.slice/crio-d0c74d8f947ef3548437b498c1319c3890436f3c5063185580a6ab03b0ff2629 WatchSource:0}: Error finding container d0c74d8f947ef3548437b498c1319c3890436f3c5063185580a6ab03b0ff2629: Status 404 returned error can't find the container with id d0c74d8f947ef3548437b498c1319c3890436f3c5063185580a6ab03b0ff2629
Jul 27 21:03:00.193793 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:00.193774    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:03:01.147932 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:01.147898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx" event={"ID":"83986328-9388-4556-b86f-b971c9d17d98","Type":"ContainerStarted","Data":"d0c74d8f947ef3548437b498c1319c3890436f3c5063185580a6ab03b0ff2629"}
Jul 27 21:03:03.159468 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:03.159435    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx" event={"ID":"83986328-9388-4556-b86f-b971c9d17d98","Type":"ContainerStarted","Data":"97ccdc4113c143b63f6f59ff354dfe5fa1389b537d561d1b24166dd4401b4059"}
Jul 27 21:03:03.179504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:03.179451    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx" podStartSLOduration=2.142872883 podStartE2EDuration="4.179424806s" totalImagesPullingTime="2.036551923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:03.177581389 +0000 UTC m=+688.951513809" watchObservedRunningTime="2026-07-27 21:03:03.179424806 +0000 UTC m=+688.953357226"
Jul 27 21:03:07.931144 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:07.931098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:10.044854 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:10.044809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:10.045370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:10.044887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:10.057610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:10.057583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:10.203831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:10.203803    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ql9qx"
Jul 27 21:03:16.220723 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:16.220585    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:18.912810 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.912775    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"]
Jul 27 21:03:18.916498 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.916478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.919466 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.919444    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 27 21:03:18.932546 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.932518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"]
Jul 27 21:03:18.974345 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.974310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:18.993370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.993536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.993536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.993536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzhn\" (UniqueName: \"kubernetes.io/projected/dd1b7350-a42d-4da0-ba85-68a9b84443fc-kube-api-access-6nzhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.993536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:18.993536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:18.993498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.093902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.093866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.093902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.093902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzhn\" (UniqueName: \"kubernetes.io/projected/dd1b7350-a42d-4da0-ba85-68a9b84443fc-kube-api-access-6nzhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094134 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094134 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094351 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094402 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094483 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094463    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.094540 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.094495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.096595 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.096569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1b7350-a42d-4da0-ba85-68a9b84443fc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.096735 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.096718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1b7350-a42d-4da0-ba85-68a9b84443fc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.103649 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.103623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzhn\" (UniqueName: \"kubernetes.io/projected/dd1b7350-a42d-4da0-ba85-68a9b84443fc-kube-api-access-6nzhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5cskg\" (UID: \"dd1b7350-a42d-4da0-ba85-68a9b84443fc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.227125 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.227086    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:19.373684 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:19.373605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"]
Jul 27 21:03:19.376074 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:03:19.376042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1b7350_a42d_4da0_ba85_68a9b84443fc.slice/crio-2745b77fe00f7d6cda25e38d2272adb85a9d92bb8a115051d216590db8b00b61 WatchSource:0}: Error finding container 2745b77fe00f7d6cda25e38d2272adb85a9d92bb8a115051d216590db8b00b61: Status 404 returned error can't find the container with id 2745b77fe00f7d6cda25e38d2272adb85a9d92bb8a115051d216590db8b00b61
Jul 27 21:03:20.242711 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:20.242680    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg" event={"ID":"dd1b7350-a42d-4da0-ba85-68a9b84443fc","Type":"ContainerStarted","Data":"01a3d5d731badf009a7c6cf17dcc33eb9fb761e33878bca24e17fd0fd13941e7"}
Jul 27 21:03:20.243087 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:20.242733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg" event={"ID":"dd1b7350-a42d-4da0-ba85-68a9b84443fc","Type":"ContainerStarted","Data":"2745b77fe00f7d6cda25e38d2272adb85a9d92bb8a115051d216590db8b00b61"}
Jul 27 21:03:20.266239 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:20.266176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg" podStartSLOduration=2.065112432 podStartE2EDuration="2.26616062s" totalImagesPullingTime="201.048188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:20.263867114 +0000 UTC m=+706.037799533" watchObservedRunningTime="2026-07-27 21:03:20.26616062 +0000 UTC m=+706.040093055"
Jul 27 21:03:21.679190 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:21.679156    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:24.390841 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.390811    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"]
Jul 27 21:03:24.394507 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.394491    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.396946 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.396924    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 21:03:24.405090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.405067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"]
Jul 27 21:03:24.446373 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbdx\" (UniqueName: \"kubernetes.io/projected/22a2be61-e35c-425f-825d-a4462ea0a390-kube-api-access-7qbdx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.446527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.446527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.446527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a2be61-e35c-425f-825d-a4462ea0a390-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.446527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.446677 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.446534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547334 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547301    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbdx\" (UniqueName: \"kubernetes.io/projected/22a2be61-e35c-425f-825d-a4462ea0a390-kube-api-access-7qbdx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a2be61-e35c-425f-825d-a4462ea0a390-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547858 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547826    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547967 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.547967 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.547897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.549694 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.549674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22a2be61-e35c-425f-825d-a4462ea0a390-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.550001 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.549986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a2be61-e35c-425f-825d-a4462ea0a390-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.555283 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.555263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbdx\" (UniqueName: \"kubernetes.io/projected/22a2be61-e35c-425f-825d-a4462ea0a390-kube-api-access-7qbdx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj\" (UID: \"22a2be61-e35c-425f-825d-a4462ea0a390\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.574828 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.574800    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:24.716770 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.716735    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:24.857699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:24.857674    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"]
Jul 27 21:03:24.859930 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:03:24.859896    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22a2be61_e35c_425f_825d_a4462ea0a390.slice/crio-54a8ee340d29eb7fb2ce717f17c34e9ee5e453a2f7c080ab5e18204b97ea3b08 WatchSource:0}: Error finding container 54a8ee340d29eb7fb2ce717f17c34e9ee5e453a2f7c080ab5e18204b97ea3b08: Status 404 returned error can't find the container with id 54a8ee340d29eb7fb2ce717f17c34e9ee5e453a2f7c080ab5e18204b97ea3b08
Jul 27 21:03:25.267797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:25.267763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj" event={"ID":"22a2be61-e35c-425f-825d-a4462ea0a390","Type":"ContainerStarted","Data":"1beccacdca33eef0a4de37731bb377635e67e09708885e8fa357a0eea00eb5ba"}
Jul 27 21:03:25.267797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:25.267801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj" event={"ID":"22a2be61-e35c-425f-825d-a4462ea0a390","Type":"ContainerStarted","Data":"54a8ee340d29eb7fb2ce717f17c34e9ee5e453a2f7c080ab5e18204b97ea3b08"}
Jul 27 21:03:25.286446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:25.286387    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj" podStartSLOduration=1.089131608 podStartE2EDuration="1.286369014s" totalImagesPullingTime="197.237406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:03:25.284190118 +0000 UTC m=+711.058122539" watchObservedRunningTime="2026-07-27 21:03:25.286369014 +0000 UTC m=+711.060301434"
Jul 27 21:03:29.227869 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:29.227827    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:29.228510 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:29.227905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:29.240850 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:29.240819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:29.298003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:29.297966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5cskg"
Jul 27 21:03:30.732774 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:30.732735    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:34.717629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:34.717596    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:34.718053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:34.717644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:34.731592 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:34.731564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:35.329791 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:35.329762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj"
Jul 27 21:03:39.467683 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:39.467648    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:39.569896 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:03:39.569859    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:08.643096 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:08.643059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:08.742957 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:08.742922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:09.874650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:09.874616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:09.970658 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:09.970622    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:10.993030 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:10.992989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:56.402348 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.402261    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:04:56.403065 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.402492    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" podUID="85edd041-2fc1-4a03-a1d1-da900a44151e" containerName="manager" containerID="cri-o://7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607" gracePeriod=30
Jul 27 21:04:56.655347 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.655281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:04:56.681845 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.681811    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert\") pod \"85edd041-2fc1-4a03-a1d1-da900a44151e\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") "
Jul 27 21:04:56.682044 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.681929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qf9\" (UniqueName: \"kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9\") pod \"85edd041-2fc1-4a03-a1d1-da900a44151e\" (UID: \"85edd041-2fc1-4a03-a1d1-da900a44151e\") "
Jul 27 21:04:56.684778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.684743    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "85edd041-2fc1-4a03-a1d1-da900a44151e" (UID: "85edd041-2fc1-4a03-a1d1-da900a44151e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:04:56.684930 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.684812    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9" (OuterVolumeSpecName: "kube-api-access-b9qf9") pod "85edd041-2fc1-4a03-a1d1-da900a44151e" (UID: "85edd041-2fc1-4a03-a1d1-da900a44151e"). InnerVolumeSpecName "kube-api-access-b9qf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:04:56.711333 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.711300    2580 generic.go:356] "Generic (PLEG): container finished" podID="85edd041-2fc1-4a03-a1d1-da900a44151e" containerID="7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607" exitCode=0
Jul 27 21:04:56.711514 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.711347    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-2p6jb"
Jul 27 21:04:56.711514 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.711377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" event={"ID":"85edd041-2fc1-4a03-a1d1-da900a44151e","Type":"ContainerDied","Data":"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"}
Jul 27 21:04:56.711514 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.711422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-2p6jb" event={"ID":"85edd041-2fc1-4a03-a1d1-da900a44151e","Type":"ContainerDied","Data":"b3c585e58b9096df86adef7533e4f2ee512c91a367996c7cc45a316ca734347a"}
Jul 27 21:04:56.711514 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.711442    2580 scope.go:117] "RemoveContainer" containerID="7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"
Jul 27 21:04:56.721175 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.721155    2580 scope.go:117] "RemoveContainer" containerID="7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"
Jul 27 21:04:56.721594 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:04:56.721566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607\": container with ID starting with 7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607 not found: ID does not exist" containerID="7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"
Jul 27 21:04:56.721674 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.721598    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607"} err="failed to get container status \"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607\": rpc error: code = NotFound desc = could not find container \"7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607\": container with ID starting with 7c8cfe774897aa0865f782727115ce5f1b2bdd370ad1cb34d9322c6a73f4f607 not found: ID does not exist"
Jul 27 21:04:56.735148 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.735116    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:04:56.741439 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.741410    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-2p6jb"]
Jul 27 21:04:56.783328 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.783295    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85edd041-2fc1-4a03-a1d1-da900a44151e-webhook-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:04:56.783328 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.783323    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qf9\" (UniqueName: \"kubernetes.io/projected/85edd041-2fc1-4a03-a1d1-da900a44151e-kube-api-access-b9qf9\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:04:56.805819 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:04:56.805792    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85edd041-2fc1-4a03-a1d1-da900a44151e" path="/var/lib/kubelet/pods/85edd041-2fc1-4a03-a1d1-da900a44151e/volumes"
Jul 27 21:05:57.576173 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.576136    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:05:57.576661 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.576538    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85edd041-2fc1-4a03-a1d1-da900a44151e" containerName="manager"
Jul 27 21:05:57.576661 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.576550    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="85edd041-2fc1-4a03-a1d1-da900a44151e" containerName="manager"
Jul 27 21:05:57.576661 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.576605    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="85edd041-2fc1-4a03-a1d1-da900a44151e" containerName="manager"
Jul 27 21:05:57.580456 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.580438    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.583035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.583014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zdvqq\""
Jul 27 21:05:57.583132 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.583015    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:05:57.584382 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.584365    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:05:57.588722 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.588703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:05:57.644287 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.644245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.644484 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.644299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvp2\" (UniqueName: \"kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.745492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.745455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.745492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.745496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvp2\" (UniqueName: \"kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.748034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.748001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.753833 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.753804    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvp2\" (UniqueName: \"kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2\") pod \"maas-controller-55f67d7fff-qhnnm\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") " pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:57.892383 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:57.892263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:58.038020 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:58.037974    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:05:58.040646 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:05:58.040614    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942bd4aa_51e5_4d56_b393_acaabed3918d.slice/crio-d12e224211f675d9f01df7767116bea02fa4626ea2d665c8e1526760e76771ba WatchSource:0}: Error finding container d12e224211f675d9f01df7767116bea02fa4626ea2d665c8e1526760e76771ba: Status 404 returned error can't find the container with id d12e224211f675d9f01df7767116bea02fa4626ea2d665c8e1526760e76771ba
Jul 27 21:05:59.013512 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:59.013463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" event={"ID":"942bd4aa-51e5-4d56-b393-acaabed3918d","Type":"ContainerStarted","Data":"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"}
Jul 27 21:05:59.013912 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:59.013550    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" event={"ID":"942bd4aa-51e5-4d56-b393-acaabed3918d","Type":"ContainerStarted","Data":"d12e224211f675d9f01df7767116bea02fa4626ea2d665c8e1526760e76771ba"}
Jul 27 21:05:59.013912 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:59.013581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:05:59.033571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:05:59.033508    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" podStartSLOduration=1.5750041019999999 podStartE2EDuration="2.033490553s" totalImagesPullingTime="458.486451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:59.031510039 +0000 UTC m=+864.805442462" watchObservedRunningTime="2026-07-27 21:05:59.033490553 +0000 UTC m=+864.807422974"
Jul 27 21:06:10.021620 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:10.021589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:06:14.582882 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:14.582845    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:14.667592 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:14.667549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:06:14.667789 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:14.667764    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" podUID="942bd4aa-51e5-4d56-b393-acaabed3918d" containerName="manager" containerID="cri-o://ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9" gracePeriod=30
Jul 27 21:06:14.929363 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:14.929338    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:06:15.008352 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.008315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtvp2\" (UniqueName: \"kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2\") pod \"942bd4aa-51e5-4d56-b393-acaabed3918d\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") "
Jul 27 21:06:15.008554 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.008430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert\") pod \"942bd4aa-51e5-4d56-b393-acaabed3918d\" (UID: \"942bd4aa-51e5-4d56-b393-acaabed3918d\") "
Jul 27 21:06:15.010628 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.010600    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2" (OuterVolumeSpecName: "kube-api-access-xtvp2") pod "942bd4aa-51e5-4d56-b393-acaabed3918d" (UID: "942bd4aa-51e5-4d56-b393-acaabed3918d"). InnerVolumeSpecName "kube-api-access-xtvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:15.010713 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.010660    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "942bd4aa-51e5-4d56-b393-acaabed3918d" (UID: "942bd4aa-51e5-4d56-b393-acaabed3918d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:15.092113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.092025    2580 generic.go:356] "Generic (PLEG): container finished" podID="942bd4aa-51e5-4d56-b393-acaabed3918d" containerID="ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9" exitCode=0
Jul 27 21:06:15.092113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.092068    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-qhnnm"
Jul 27 21:06:15.092113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.092107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" event={"ID":"942bd4aa-51e5-4d56-b393-acaabed3918d","Type":"ContainerDied","Data":"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"}
Jul 27 21:06:15.092352 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.092147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-qhnnm" event={"ID":"942bd4aa-51e5-4d56-b393-acaabed3918d","Type":"ContainerDied","Data":"d12e224211f675d9f01df7767116bea02fa4626ea2d665c8e1526760e76771ba"}
Jul 27 21:06:15.092352 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.092168    2580 scope.go:117] "RemoveContainer" containerID="ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"
Jul 27 21:06:15.101752 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.101727    2580 scope.go:117] "RemoveContainer" containerID="ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"
Jul 27 21:06:15.102068 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:06:15.102051    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9\": container with ID starting with ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9 not found: ID does not exist" containerID="ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"
Jul 27 21:06:15.102117 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.102080    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9"} err="failed to get container status \"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9\": rpc error: code = NotFound desc = could not find container \"ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9\": container with ID starting with ecbd2541295d2f0624c14f44573845295e5043806834019dccc4dc9c8704cac9 not found: ID does not exist"
Jul 27 21:06:15.109955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.109929    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvp2\" (UniqueName: \"kubernetes.io/projected/942bd4aa-51e5-4d56-b393-acaabed3918d-kube-api-access-xtvp2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:06:15.109955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.109956    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/942bd4aa-51e5-4d56-b393-acaabed3918d-webhook-cert\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:06:15.112959 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.112932    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:06:15.116689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.116665    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-qhnnm"]
Jul 27 21:06:15.797260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.797202    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:15.910536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.910499    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55f67d7fff-6p9ql"]
Jul 27 21:06:15.910882 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.910870    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942bd4aa-51e5-4d56-b393-acaabed3918d" containerName="manager"
Jul 27 21:06:15.910937 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.910884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="942bd4aa-51e5-4d56-b393-acaabed3918d" containerName="manager"
Jul 27 21:06:15.910973 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.910940    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="942bd4aa-51e5-4d56-b393-acaabed3918d" containerName="manager"
Jul 27 21:06:15.915403 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.915386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:15.917834 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.917815    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 27 21:06:15.918890 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.918866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zdvqq\""
Jul 27 21:06:15.919000 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.918922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 27 21:06:15.922403 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:15.922379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-6p9ql"]
Jul 27 21:06:16.018291 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.018251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/783e4803-42ff-47e9-b2c8-25dafefec617-webhook-cert\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.018475 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.018399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpp4\" (UniqueName: \"kubernetes.io/projected/783e4803-42ff-47e9-b2c8-25dafefec617-kube-api-access-pfpp4\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.119119 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.119029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpp4\" (UniqueName: \"kubernetes.io/projected/783e4803-42ff-47e9-b2c8-25dafefec617-kube-api-access-pfpp4\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.119119 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.119085    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/783e4803-42ff-47e9-b2c8-25dafefec617-webhook-cert\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.121809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.121786    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/783e4803-42ff-47e9-b2c8-25dafefec617-webhook-cert\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.136607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.136575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpp4\" (UniqueName: \"kubernetes.io/projected/783e4803-42ff-47e9-b2c8-25dafefec617-kube-api-access-pfpp4\") pod \"maas-controller-55f67d7fff-6p9ql\" (UID: \"783e4803-42ff-47e9-b2c8-25dafefec617\") " pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.227836 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.227782    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:16.375932 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.375907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f67d7fff-6p9ql"]
Jul 27 21:06:16.378691 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:06:16.378658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783e4803_42ff_47e9_b2c8_25dafefec617.slice/crio-75b234e3563e50c2705accb7939e451b13e17292475ecbe6bbd0b4ce9e0ceb67 WatchSource:0}: Error finding container 75b234e3563e50c2705accb7939e451b13e17292475ecbe6bbd0b4ce9e0ceb67: Status 404 returned error can't find the container with id 75b234e3563e50c2705accb7939e451b13e17292475ecbe6bbd0b4ce9e0ceb67
Jul 27 21:06:16.804924 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:16.804890    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942bd4aa-51e5-4d56-b393-acaabed3918d" path="/var/lib/kubelet/pods/942bd4aa-51e5-4d56-b393-acaabed3918d/volumes"
Jul 27 21:06:17.104089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:17.103988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-6p9ql" event={"ID":"783e4803-42ff-47e9-b2c8-25dafefec617","Type":"ContainerStarted","Data":"31bdaff6989034c2e460a92b7c5db2bc0a8b33746a0150d1771c23f1ab5966a3"}
Jul 27 21:06:17.104089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:17.104067    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f67d7fff-6p9ql" event={"ID":"783e4803-42ff-47e9-b2c8-25dafefec617","Type":"ContainerStarted","Data":"75b234e3563e50c2705accb7939e451b13e17292475ecbe6bbd0b4ce9e0ceb67"}
Jul 27 21:06:17.104310 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:17.104106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:17.123559 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:17.123495    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55f67d7fff-6p9ql" podStartSLOduration=1.67609661 podStartE2EDuration="2.123476467s" totalImagesPullingTime="447.379857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:17.123092997 +0000 UTC m=+882.897025419" watchObservedRunningTime="2026-07-27 21:06:17.123476467 +0000 UTC m=+882.897408889"
Jul 27 21:06:28.113776 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:28.113700    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55f67d7fff-6p9ql"
Jul 27 21:06:35.597556 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:35.597502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:06:35.597556 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:35.597500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:06:56.842124 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:56.842083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:58.112391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:06:58.112349    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:02.734013 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:02.733974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:02.840380 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:02.840322    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:02.935680 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:02.935649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:03.209066 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:03.209031    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:08.824076 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:08.824039    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:08.891560 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:07:08.891523    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:27.955346 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:08:27.955304    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:18.546257 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:09:18.546203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:35.625916 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:11:35.625888    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:11:35.627363 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:11:35.627335    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:11:57.160228 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:11:57.160182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:21.813450 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:12:21.813413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:23.555417 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:12:23.555340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:58.107114 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.107079    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:14:58.110786 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.110765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.113306 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.113288    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fd2c4ccc-kserve-self-signed-certs\""
Jul 27 21:14:58.125397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.125361    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:14:58.167968 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.167932    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:58.218506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.218695 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.218695 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218543    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4w6\" (UniqueName: \"kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.218695 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218611    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.218695 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.218695 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.218657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320100 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320100 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320105    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320123    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320180    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4w6\" (UniqueName: \"kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.320835 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.320771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.322768 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.322737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.322929 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.322908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.328169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.328141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4w6\" (UniqueName: \"kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6\") pod \"e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") " pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.420150 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.420054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:14:58.562001 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.561975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:14:58.564067 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:14:58.564039    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881ccfd9_d1f4_44c1_b3d6_519650a69026.slice/crio-78ab2e5476eba80f5140f5b60ac4ac2afd30df8d15398e014db523bf4957e728 WatchSource:0}: Error finding container 78ab2e5476eba80f5140f5b60ac4ac2afd30df8d15398e014db523bf4957e728: Status 404 returned error can't find the container with id 78ab2e5476eba80f5140f5b60ac4ac2afd30df8d15398e014db523bf4957e728
Jul 27 21:14:58.611198 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:58.611165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" event={"ID":"881ccfd9-d1f4-44c1-b3d6-519650a69026","Type":"ContainerStarted","Data":"78ab2e5476eba80f5140f5b60ac4ac2afd30df8d15398e014db523bf4957e728"}
Jul 27 21:14:59.619977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:59.619936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" event={"ID":"881ccfd9-d1f4-44c1-b3d6-519650a69026","Type":"ContainerStarted","Data":"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"}
Jul 27 21:14:59.640345 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:14:59.640285    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" podStartSLOduration=1.640264801 podStartE2EDuration="1.640264801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:59.638790386 +0000 UTC m=+1405.412722797" watchObservedRunningTime="2026-07-27 21:14:59.640264801 +0000 UTC m=+1405.414197222"
Jul 27 21:15:08.420281 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:08.420185    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:08.420281 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:08.420282    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:08.433166 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:08.433124    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:08.679483 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:08.679397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:08.902839 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:08.902786    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:39.974975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:39.974937    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:15:39.995876 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:39.995846    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:40.714819 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:40.714789    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:54.995387 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:54.995318    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" podUID="881ccfd9-d1f4-44c1-b3d6-519650a69026" containerName="main" containerID="cri-o://01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea" gracePeriod=45
Jul 27 21:15:55.262943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.262908    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:55.350483 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350667 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350498    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v4w6\" (UniqueName: \"kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350667 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350667 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350667 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350905 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm\") pod \"881ccfd9-d1f4-44c1-b3d6-519650a69026\" (UID: \"881ccfd9-d1f4-44c1-b3d6-519650a69026\") "
Jul 27 21:15:55.350965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350876    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:55.351018 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home" (OuterVolumeSpecName: "home") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:55.351074 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.351032    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.351127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.350790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache" (OuterVolumeSpecName: "model-cache") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:55.353337 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.353297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm" (OuterVolumeSpecName: "dshm") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:55.353474 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.353399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6" (OuterVolumeSpecName: "kube-api-access-7v4w6") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "kube-api-access-7v4w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:15:55.353533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.353505    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "881ccfd9-d1f4-44c1-b3d6-519650a69026" (UID: "881ccfd9-d1f4-44c1-b3d6-519650a69026"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:15:55.452492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.452453    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/881ccfd9-d1f4-44c1-b3d6-519650a69026-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.452492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.452484    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.452492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.452494    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.452492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.452502    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/881ccfd9-d1f4-44c1-b3d6-519650a69026-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.452769 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.452511    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v4w6\" (UniqueName: \"kubernetes.io/projected/881ccfd9-d1f4-44c1-b3d6-519650a69026-kube-api-access-7v4w6\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:15:55.899723 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.899687    2580 generic.go:356] "Generic (PLEG): container finished" podID="881ccfd9-d1f4-44c1-b3d6-519650a69026" containerID="01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea" exitCode=0
Jul 27 21:15:55.899894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.899729    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"
Jul 27 21:15:55.899894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.899771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" event={"ID":"881ccfd9-d1f4-44c1-b3d6-519650a69026","Type":"ContainerDied","Data":"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"}
Jul 27 21:15:55.899894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.899818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs" event={"ID":"881ccfd9-d1f4-44c1-b3d6-519650a69026","Type":"ContainerDied","Data":"78ab2e5476eba80f5140f5b60ac4ac2afd30df8d15398e014db523bf4957e728"}
Jul 27 21:15:55.899894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.899836    2580 scope.go:117] "RemoveContainer" containerID="01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"
Jul 27 21:15:55.909538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.909519    2580 scope.go:117] "RemoveContainer" containerID="01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"
Jul 27 21:15:55.909874 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:15:55.909853    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea\": container with ID starting with 01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea not found: ID does not exist" containerID="01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"
Jul 27 21:15:55.909935 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.909887    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea"} err="failed to get container status \"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea\": rpc error: code = NotFound desc = could not find container \"01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea\": container with ID starting with 01d4cc9caa28c036eb41b10aa19a63173a0058598736e783874db8c97fb4c2ea not found: ID does not exist"
Jul 27 21:15:55.920034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.920007    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:15:55.931868 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:55.928080    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fd2c4ccc-kserve-547cb7c4c9-czjqs"]
Jul 27 21:15:56.805436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:15:56.805399    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881ccfd9-d1f4-44c1-b3d6-519650a69026" path="/var/lib/kubelet/pods/881ccfd9-d1f4-44c1-b3d6-519650a69026/volumes"
Jul 27 21:16:35.662974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:16:35.662942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:16:35.665102 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:16:35.665080    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:16:49.179614 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:16:49.179545    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:47.239884 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:18:47.239837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:06.536192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:06.536138    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:09.654775 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:09.654730    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:09.750780 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:09.750738    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:10.120509 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:10.120474    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:15.882473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:15.882431    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:15.941506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:15.941468    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:16.114423 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:16.114386    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:16.426562 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:16.426525    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:16.766350 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:16.766315    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.265490 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.265449    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.339794 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.339756    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:19.340163 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.340150    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881ccfd9-d1f4-44c1-b3d6-519650a69026" containerName="main"
Jul 27 21:19:19.340208 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.340165    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="881ccfd9-d1f4-44c1-b3d6-519650a69026" containerName="main"
Jul 27 21:19:19.340285 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.340241    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="881ccfd9-d1f4-44c1-b3d6-519650a69026" containerName="main"
Jul 27 21:19:19.343284 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.343267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.345697 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.345674    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-39c7c327-serving-cert\""
Jul 27 21:19:19.355076 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.355043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:19.359455 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.359425    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.423518 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.423481    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.425512 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.425492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.425573 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.425519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4nh\" (UniqueName: \"kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.425643 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.425625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.526472 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.526374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.526472 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.526429    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.526693 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:19.526528    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-serving-cert: secret "maas-api-e2e-ait-39c7c327-serving-cert" not found
Jul 27 21:19:19.526693 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.526547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4nh\" (UniqueName: \"kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.526693 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:19.526596    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls podName:65c46789-fc05-4205-a970-c257f00ee137 nodeName:}" failed. No retries permitted until 2026-07-27 21:19:20.02657927 +0000 UTC m=+1665.800511667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls") pod "maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" (UID: "65c46789-fc05-4205-a970-c257f00ee137") : secret "maas-api-e2e-ait-39c7c327-serving-cert" not found
Jul 27 21:19:19.527110 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.527091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.537044 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.537013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4nh\" (UniqueName: \"kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:19.841126 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.841028    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.935981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.935939    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:19.980844 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.980807    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:19.984372 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.984356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:19.987280 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.987258    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-39c7c327\""
Jul 27 21:19:19.987404 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.987331    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-39c7c327-dockercfg-t27vm\""
Jul 27 21:19:19.993036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:19.993010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:20.014940 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.014909    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:20.018393 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.018378    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.031104 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.031084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:20.031516 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.031490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbs85\" (UniqueName: \"kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.031631 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.031549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:20.031631 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.031577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.034306 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.034283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") pod \"maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:20.132886 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.132774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.132886 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.132831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqq5\" (UniqueName: \"kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.133126 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.132952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.133126 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.133089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbs85\" (UniqueName: \"kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.133570 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.133550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.141570 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.141545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbs85\" (UniqueName: \"kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85\") pod \"payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") " pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.234184 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.234135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqq5\" (UniqueName: \"kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.234184 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.234189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.234961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.234936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.243119 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.243088    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqq5\" (UniqueName: \"kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5\") pod \"payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.254859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.254824    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:20.295961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.295926    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.328512 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.328456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.424978 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.424943    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:20.426324 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:19:20.426290    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c46789_fc05_4205_a970_c257f00ee137.slice/crio-05684cfd1a0218211a376ae678027ad71906efa855ea08ed98403c412be59cdd WatchSource:0}: Error finding container 05684cfd1a0218211a376ae678027ad71906efa855ea08ed98403c412be59cdd: Status 404 returned error can't find the container with id 05684cfd1a0218211a376ae678027ad71906efa855ea08ed98403c412be59cdd
Jul 27 21:19:20.427868 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.427850    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:19:20.471246 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.471184    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:20.474351 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:19:20.474320    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ad454f_702c_4660_8c1b_b4b92580198e.slice/crio-4c40a405bfeef0aadec11eec689f247c65e0a9d275c32148b2d8093158e3741e WatchSource:0}: Error finding container 4c40a405bfeef0aadec11eec689f247c65e0a9d275c32148b2d8093158e3741e: Status 404 returned error can't find the container with id 4c40a405bfeef0aadec11eec689f247c65e0a9d275c32148b2d8093158e3741e
Jul 27 21:19:20.497961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.497937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:20.509386 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:19:20.509356    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104435e5_6908_41db_9bd8_9da9ca601dfd.slice/crio-4a1e8d0ad256908dabeb384ffef8b774ea1c434645314d59b790c7b55e300135 WatchSource:0}: Error finding container 4a1e8d0ad256908dabeb384ffef8b774ea1c434645314d59b790c7b55e300135: Status 404 returned error can't find the container with id 4a1e8d0ad256908dabeb384ffef8b774ea1c434645314d59b790c7b55e300135
Jul 27 21:19:20.880547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.880507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" event={"ID":"104435e5-6908-41db-9bd8-9da9ca601dfd","Type":"ContainerStarted","Data":"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"}
Jul 27 21:19:20.880740 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.880579    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:20.880740 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.880606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" event={"ID":"104435e5-6908-41db-9bd8-9da9ca601dfd","Type":"ContainerStarted","Data":"4a1e8d0ad256908dabeb384ffef8b774ea1c434645314d59b790c7b55e300135"}
Jul 27 21:19:20.882267 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.882245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" event={"ID":"e9ad454f-702c-4660-8c1b-b4b92580198e","Type":"ContainerStarted","Data":"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"}
Jul 27 21:19:20.882398 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.882289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" event={"ID":"e9ad454f-702c-4660-8c1b-b4b92580198e","Type":"ContainerStarted","Data":"4c40a405bfeef0aadec11eec689f247c65e0a9d275c32148b2d8093158e3741e"}
Jul 27 21:19:20.882398 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.882313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:20.883324 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.883303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" event={"ID":"65c46789-fc05-4205-a970-c257f00ee137","Type":"ContainerStarted","Data":"05684cfd1a0218211a376ae678027ad71906efa855ea08ed98403c412be59cdd"}
Jul 27 21:19:20.899367 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.899310    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" podStartSLOduration=1.899291028 podStartE2EDuration="1.899291028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:20.896058326 +0000 UTC m=+1666.669990749" watchObservedRunningTime="2026-07-27 21:19:20.899291028 +0000 UTC m=+1666.673223452"
Jul 27 21:19:20.915694 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:20.915642    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" podStartSLOduration=1.9156294969999998 podStartE2EDuration="1.915629497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:20.914458951 +0000 UTC m=+1666.688391372" watchObservedRunningTime="2026-07-27 21:19:20.915629497 +0000 UTC m=+1666.689561960"
Jul 27 21:19:21.892031 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:21.891986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" event={"ID":"65c46789-fc05-4205-a970-c257f00ee137","Type":"ContainerStarted","Data":"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"}
Jul 27 21:19:21.892471 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:21.892151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:21.910382 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:21.910334    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" podStartSLOduration=2.373085269 podStartE2EDuration="2.910319159s" totalImagesPullingTime="537.23389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:21.907967564 +0000 UTC m=+1667.681899984" watchObservedRunningTime="2026-07-27 21:19:21.910319159 +0000 UTC m=+1667.684251579"
Jul 27 21:19:25.156630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:25.156588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:27.902223 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:27.902193    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:31.894449 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:31.894416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:31.894927 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:31.894493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:55.237977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.237897    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:55.238524 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.238108    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" podUID="65c46789-fc05-4205-a970-c257f00ee137" containerName="maas-api" containerID="cri-o://64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c" gracePeriod=30
Jul 27 21:19:55.260118 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.260080    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:55.283003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.282969    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:55.283234 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.283190    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" podUID="e9ad454f-702c-4660-8c1b-b4b92580198e" containerName="payload-processing" containerID="cri-o://892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2" gracePeriod=30
Jul 27 21:19:55.288361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.288331    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:55.288563 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.288543    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" podUID="104435e5-6908-41db-9bd8-9da9ca601dfd" containerName="payload-pre-processing" containerID="cri-o://252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877" gracePeriod=30
Jul 27 21:19:55.322140 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.322111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:55.380754 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:55.380716    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ad454f_702c_4660_8c1b_b4b92580198e.slice/crio-conmon-892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ad454f_702c_4660_8c1b_b4b92580198e.slice/crio-892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 21:19:55.407705 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.407460    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:55.548739 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.548438    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:55.563546 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.563519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume\") pod \"e9ad454f-702c-4660-8c1b-b4b92580198e\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") "
Jul 27 21:19:55.563675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.563562    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbs85\" (UniqueName: \"kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85\") pod \"e9ad454f-702c-4660-8c1b-b4b92580198e\" (UID: \"e9ad454f-702c-4660-8c1b-b4b92580198e\") "
Jul 27 21:19:55.563900 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.563877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9ad454f-702c-4660-8c1b-b4b92580198e" (UID: "e9ad454f-702c-4660-8c1b-b4b92580198e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:55.566405 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.566368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85" (OuterVolumeSpecName: "kube-api-access-gbs85") pod "e9ad454f-702c-4660-8c1b-b4b92580198e" (UID: "e9ad454f-702c-4660-8c1b-b4b92580198e"). InnerVolumeSpecName "kube-api-access-gbs85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:55.567286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.567249    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:55.596707 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.596678    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:55.664046 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664011    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") pod \"65c46789-fc05-4205-a970-c257f00ee137\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") "
Jul 27 21:19:55.664199 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume\") pod \"104435e5-6908-41db-9bd8-9da9ca601dfd\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") "
Jul 27 21:19:55.664199 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz4nh\" (UniqueName: \"kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh\") pod \"65c46789-fc05-4205-a970-c257f00ee137\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") "
Jul 27 21:19:55.664199 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664180    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca\") pod \"65c46789-fc05-4205-a970-c257f00ee137\" (UID: \"65c46789-fc05-4205-a970-c257f00ee137\") "
Jul 27 21:19:55.664423 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkqq5\" (UniqueName: \"kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5\") pod \"104435e5-6908-41db-9bd8-9da9ca601dfd\" (UID: \"104435e5-6908-41db-9bd8-9da9ca601dfd\") "
Jul 27 21:19:55.664487 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664457    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ad454f-702c-4660-8c1b-b4b92580198e-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.664487 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664475    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbs85\" (UniqueName: \"kubernetes.io/projected/e9ad454f-702c-4660-8c1b-b4b92580198e-kube-api-access-gbs85\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.664576 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664507    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "104435e5-6908-41db-9bd8-9da9ca601dfd" (UID: "104435e5-6908-41db-9bd8-9da9ca601dfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:55.664576 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.664556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65c46789-fc05-4205-a970-c257f00ee137" (UID: "65c46789-fc05-4205-a970-c257f00ee137"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:55.666505 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.666479    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5" (OuterVolumeSpecName: "kube-api-access-vkqq5") pod "104435e5-6908-41db-9bd8-9da9ca601dfd" (UID: "104435e5-6908-41db-9bd8-9da9ca601dfd"). InnerVolumeSpecName "kube-api-access-vkqq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:55.666630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.666504    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65c46789-fc05-4205-a970-c257f00ee137" (UID: "65c46789-fc05-4205-a970-c257f00ee137"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:19:55.666630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.666612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh" (OuterVolumeSpecName: "kube-api-access-rz4nh") pod "65c46789-fc05-4205-a970-c257f00ee137" (UID: "65c46789-fc05-4205-a970-c257f00ee137"). InnerVolumeSpecName "kube-api-access-rz4nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:55.764981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.764890    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/104435e5-6908-41db-9bd8-9da9ca601dfd-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.764981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.764922    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4nh\" (UniqueName: \"kubernetes.io/projected/65c46789-fc05-4205-a970-c257f00ee137-kube-api-access-rz4nh\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.764981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.764931    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c46789-fc05-4205-a970-c257f00ee137-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.764981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.764940    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkqq5\" (UniqueName: \"kubernetes.io/projected/104435e5-6908-41db-9bd8-9da9ca601dfd-kube-api-access-vkqq5\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:55.764981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:55.764950    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65c46789-fc05-4205-a970-c257f00ee137-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:19:56.061918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.061830    2580 generic.go:356] "Generic (PLEG): container finished" podID="104435e5-6908-41db-9bd8-9da9ca601dfd" containerID="252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877" exitCode=0
Jul 27 21:19:56.061918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.061879    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"
Jul 27 21:19:56.061918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.061882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" event={"ID":"104435e5-6908-41db-9bd8-9da9ca601dfd","Type":"ContainerDied","Data":"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"}
Jul 27 21:19:56.062180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.061943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz" event={"ID":"104435e5-6908-41db-9bd8-9da9ca601dfd","Type":"ContainerDied","Data":"4a1e8d0ad256908dabeb384ffef8b774ea1c434645314d59b790c7b55e300135"}
Jul 27 21:19:56.062180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.061971    2580 scope.go:117] "RemoveContainer" containerID="252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"
Jul 27 21:19:56.063276 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.063253    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9ad454f-702c-4660-8c1b-b4b92580198e" containerID="892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2" exitCode=0
Jul 27 21:19:56.063390 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.063292    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"
Jul 27 21:19:56.063390 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.063336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" event={"ID":"e9ad454f-702c-4660-8c1b-b4b92580198e","Type":"ContainerDied","Data":"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"}
Jul 27 21:19:56.063390 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.063384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk" event={"ID":"e9ad454f-702c-4660-8c1b-b4b92580198e","Type":"ContainerDied","Data":"4c40a405bfeef0aadec11eec689f247c65e0a9d275c32148b2d8093158e3741e"}
Jul 27 21:19:56.064799 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.064776    2580 generic.go:356] "Generic (PLEG): container finished" podID="65c46789-fc05-4205-a970-c257f00ee137" containerID="64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c" exitCode=0
Jul 27 21:19:56.064920 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.064832    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"
Jul 27 21:19:56.064920 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.064855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" event={"ID":"65c46789-fc05-4205-a970-c257f00ee137","Type":"ContainerDied","Data":"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"}
Jul 27 21:19:56.064920 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.064892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr" event={"ID":"65c46789-fc05-4205-a970-c257f00ee137","Type":"ContainerDied","Data":"05684cfd1a0218211a376ae678027ad71906efa855ea08ed98403c412be59cdd"}
Jul 27 21:19:56.071986 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.071972    2580 scope.go:117] "RemoveContainer" containerID="252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"
Jul 27 21:19:56.072281 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:56.072253    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877\": container with ID starting with 252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877 not found: ID does not exist" containerID="252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"
Jul 27 21:19:56.072367 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.072292    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877"} err="failed to get container status \"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877\": rpc error: code = NotFound desc = could not find container \"252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877\": container with ID starting with 252d1d6bc0d071c1ae08b86633be6b4bd2349e5cd676242cb70021fcaadd1877 not found: ID does not exist"
Jul 27 21:19:56.072367 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.072315    2580 scope.go:117] "RemoveContainer" containerID="892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"
Jul 27 21:19:56.081078 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.081064    2580 scope.go:117] "RemoveContainer" containerID="892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"
Jul 27 21:19:56.081362 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:56.081345    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2\": container with ID starting with 892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2 not found: ID does not exist" containerID="892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"
Jul 27 21:19:56.081408 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.081369    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2"} err="failed to get container status \"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2\": rpc error: code = NotFound desc = could not find container \"892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2\": container with ID starting with 892c81b7c2f6b9d7486663610b18f7b82026def76bd1e1a99b56a5812ea71ce2 not found: ID does not exist"
Jul 27 21:19:56.081408 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.081385    2580 scope.go:117] "RemoveContainer" containerID="64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"
Jul 27 21:19:56.085580 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.085561    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:56.089569 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.089552    2580 scope.go:117] "RemoveContainer" containerID="64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"
Jul 27 21:19:56.089812 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:19:56.089795    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c\": container with ID starting with 64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c not found: ID does not exist" containerID="64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"
Jul 27 21:19:56.089874 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.089824    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c"} err="failed to get container status \"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c\": rpc error: code = NotFound desc = could not find container \"64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c\": container with ID starting with 64f8f53df9872c6bd38f73aa9d5abd35f9144032219f1de1ce63a0eeedc5544c not found: ID does not exist"
Jul 27 21:19:56.090792 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.090776    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-39c7c327-586dcbf9cb-vr8vz"]
Jul 27 21:19:56.100503 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.100483    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:56.102730 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.102704    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-39c7c327-f5d98fdbf-mv4nr"]
Jul 27 21:19:56.113006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.112979    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:56.117090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.117072    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-39c7c327-58f56dc6bb-d8vzk"]
Jul 27 21:19:56.805077 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.805044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="104435e5-6908-41db-9bd8-9da9ca601dfd" path="/var/lib/kubelet/pods/104435e5-6908-41db-9bd8-9da9ca601dfd/volumes"
Jul 27 21:19:56.805487 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.805473    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c46789-fc05-4205-a970-c257f00ee137" path="/var/lib/kubelet/pods/65c46789-fc05-4205-a970-c257f00ee137/volumes"
Jul 27 21:19:56.805782 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:56.805770    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ad454f-702c-4660-8c1b-b4b92580198e" path="/var/lib/kubelet/pods/e9ad454f-702c-4660-8c1b-b4b92580198e/volumes"
Jul 27 21:19:59.941226 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:19:59.941175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:00.113620 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:00.113584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:00.345589 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:00.345548    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:00.640488 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:00.640403    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:00.737248 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:00.737203    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:00.843463 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:00.843430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:01.193519 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:01.193478    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:01.293328 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:01.293293    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:07.842301 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.842261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:07.960175 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:07.960565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960552    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c46789-fc05-4205-a970-c257f00ee137" containerName="maas-api"
Jul 27 21:20:07.960610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c46789-fc05-4205-a970-c257f00ee137" containerName="maas-api"
Jul 27 21:20:07.960610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="104435e5-6908-41db-9bd8-9da9ca601dfd" containerName="payload-pre-processing"
Jul 27 21:20:07.960610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960582    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="104435e5-6908-41db-9bd8-9da9ca601dfd" containerName="payload-pre-processing"
Jul 27 21:20:07.960610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960590    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ad454f-702c-4660-8c1b-b4b92580198e" containerName="payload-processing"
Jul 27 21:20:07.960610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960596    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ad454f-702c-4660-8c1b-b4b92580198e" containerName="payload-processing"
Jul 27 21:20:07.960760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c46789-fc05-4205-a970-c257f00ee137" containerName="maas-api"
Jul 27 21:20:07.960760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="104435e5-6908-41db-9bd8-9da9ca601dfd" containerName="payload-pre-processing"
Jul 27 21:20:07.960760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.960677    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ad454f-702c-4660-8c1b-b4b92580198e" containerName="payload-processing"
Jul 27 21:20:07.963548 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.963526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:07.966264 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.966237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-338bbf37-serving-cert\""
Jul 27 21:20:07.978781 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:07.978738    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:08.007437 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.007396    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.071947 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.071908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2rv2\" (UniqueName: \"kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.071947 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.071948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.072171 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.071996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.173335 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.173234    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2rv2\" (UniqueName: \"kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.173335 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.173299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.173537 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.173345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.173537 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:20:08.173449    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-serving-cert: secret "maas-api-e2e-migrate-338bbf37-serving-cert" not found
Jul 27 21:20:08.173537 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:20:08.173522    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls podName:3bddf636-11b7-4b2c-864e-0dee5d23ce04 nodeName:}" failed. No retries permitted until 2026-07-27 21:20:08.673507215 +0000 UTC m=+1714.447439613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls") pod "maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" (UID: "3bddf636-11b7-4b2c-864e-0dee5d23ce04") : secret "maas-api-e2e-migrate-338bbf37-serving-cert" not found
Jul 27 21:20:08.174025 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.174004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.191192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.191161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2rv2\" (UniqueName: \"kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.327719 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.327679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.679619 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.679581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.682329 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.682265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") pod \"maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.791886 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.791850    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.874474 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.874441    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:08.895685 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.895650    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.897223 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.897193    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:08.905793 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.905759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:08.909265 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.909237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-338bbf37\""
Jul 27 21:20:08.910169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.910139    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-338bbf37-dockercfg-gr5q6\""
Jul 27 21:20:08.911634 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.911612    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:08.924436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.924402    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:08.927936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.927910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:08.940100 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.940063    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:08.982228 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.982118    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8f9\" (UniqueName: \"kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:08.982496 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.982474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:08.982758 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.982647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qxl\" (UniqueName: \"kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:08.982758 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:08.982679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.037727 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.037089    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:09.046174 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:20:09.046133    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bddf636_11b7_4b2c_864e_0dee5d23ce04.slice/crio-49747bc9d53c05ca63215d6b8706ab8bac74dfd986b1222d66416110abfdb628 WatchSource:0}: Error finding container 49747bc9d53c05ca63215d6b8706ab8bac74dfd986b1222d66416110abfdb628: Status 404 returned error can't find the container with id 49747bc9d53c05ca63215d6b8706ab8bac74dfd986b1222d66416110abfdb628
Jul 27 21:20:09.083983 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.083934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qxl\" (UniqueName: \"kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.083983 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.083994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.084321 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.084043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8f9\" (UniqueName: \"kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:09.084321 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.084073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:09.084884 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.084860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.084970 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.084861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:09.093527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.093486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qxl\" (UniqueName: \"kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl\") pod \"payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") " pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.094404 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.094381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8f9\" (UniqueName: \"kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9\") pod \"payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:09.140061 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.140020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" event={"ID":"3bddf636-11b7-4b2c-864e-0dee5d23ce04","Type":"ContainerStarted","Data":"49747bc9d53c05ca63215d6b8706ab8bac74dfd986b1222d66416110abfdb628"}
Jul 27 21:20:09.219666 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.219630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:09.245905 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.245712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:09.382358 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.382262    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:09.385149 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:20:09.385117    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec69edc_22ef_45bb_9ba6_f9fb1ddd0e8a.slice/crio-ea30a1e6c349adcfc30ae8a6fd76a68a27541cc7b777b9c4f930b51d559b26f7 WatchSource:0}: Error finding container ea30a1e6c349adcfc30ae8a6fd76a68a27541cc7b777b9c4f930b51d559b26f7: Status 404 returned error can't find the container with id ea30a1e6c349adcfc30ae8a6fd76a68a27541cc7b777b9c4f930b51d559b26f7
Jul 27 21:20:09.416971 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:09.416946    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:09.420592 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:20:09.420566    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28352c09_054f_49cd_9d31_91c528433d45.slice/crio-5ee74536d154de2a0a1f4beb04f11567024dfa38f41795946da994bb3125c80c WatchSource:0}: Error finding container 5ee74536d154de2a0a1f4beb04f11567024dfa38f41795946da994bb3125c80c: Status 404 returned error can't find the container with id 5ee74536d154de2a0a1f4beb04f11567024dfa38f41795946da994bb3125c80c
Jul 27 21:20:10.147932 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.147892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" event={"ID":"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a","Type":"ContainerStarted","Data":"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"}
Jul 27 21:20:10.148416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.147955    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" event={"ID":"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a","Type":"ContainerStarted","Data":"ea30a1e6c349adcfc30ae8a6fd76a68a27541cc7b777b9c4f930b51d559b26f7"}
Jul 27 21:20:10.148416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.147985    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:10.150038 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.150009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" event={"ID":"3bddf636-11b7-4b2c-864e-0dee5d23ce04","Type":"ContainerStarted","Data":"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"}
Jul 27 21:20:10.150197 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.150086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:10.151688 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.151665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" event={"ID":"28352c09-054f-49cd-9d31-91c528433d45","Type":"ContainerStarted","Data":"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"}
Jul 27 21:20:10.151790 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.151703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" event={"ID":"28352c09-054f-49cd-9d31-91c528433d45","Type":"ContainerStarted","Data":"5ee74536d154de2a0a1f4beb04f11567024dfa38f41795946da994bb3125c80c"}
Jul 27 21:20:10.151870 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.151853    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:10.164612 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.164563    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" podStartSLOduration=2.16454575 podStartE2EDuration="2.16454575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:10.162503073 +0000 UTC m=+1715.936435493" watchObservedRunningTime="2026-07-27 21:20:10.16454575 +0000 UTC m=+1715.938478199"
Jul 27 21:20:10.180339 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.180286    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" podStartSLOduration=2.180267138 podStartE2EDuration="2.180267138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:10.178422879 +0000 UTC m=+1715.952355301" watchObservedRunningTime="2026-07-27 21:20:10.180267138 +0000 UTC m=+1715.954199557"
Jul 27 21:20:10.194184 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:10.194125    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" podStartSLOduration=2.690291587 podStartE2EDuration="3.194107983s" totalImagesPullingTime="503.816396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:10.19333209 +0000 UTC m=+1715.967264522" watchObservedRunningTime="2026-07-27 21:20:10.194107983 +0000 UTC m=+1715.968040449"
Jul 27 21:20:16.164038 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:16.164006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:21.159405 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:21.159348    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.46:9004: connect: connection refused"
Jul 27 21:20:21.159938 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:21.159920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:22.025742 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:22.025701    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:22.085682 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:22.085646    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:31.158657 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:31.158625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:43.240602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.240564    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:43.241081 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.240828    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" podUID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" containerName="maas-api" containerID="cri-o://c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986" gracePeriod=30
Jul 27 21:20:43.272844 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.272812    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:43.298133 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.298101    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:43.298426 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.298397    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" podUID="28352c09-054f-49cd-9d31-91c528433d45" containerName="payload-processing" containerID="cri-o://2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61" gracePeriod=30
Jul 27 21:20:43.310425 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.309890    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:43.310425 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.310092    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerName="payload-pre-processing" containerID="cri-o://4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631" gracePeriod=30
Jul 27 21:20:43.344586 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.344540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:43.424094 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.424059    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:43.571726 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.571704    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:43.575098 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.575074    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:43.600155 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.600129    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:43.699157 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5qxl\" (UniqueName: \"kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl\") pod \"28352c09-054f-49cd-9d31-91c528433d45\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca\") pod \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2rv2\" (UniqueName: \"kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2\") pod \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume\") pod \"28352c09-054f-49cd-9d31-91c528433d45\" (UID: \"28352c09-054f-49cd-9d31-91c528433d45\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume\") pod \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699347    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") pod \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\" (UID: \"3bddf636-11b7-4b2c-864e-0dee5d23ce04\") "
Jul 27 21:20:43.699381 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc8f9\" (UniqueName: \"kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9\") pod \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\" (UID: \"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a\") "
Jul 27 21:20:43.699709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3bddf636-11b7-4b2c-864e-0dee5d23ce04" (UID: "3bddf636-11b7-4b2c-864e-0dee5d23ce04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:43.699789 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699764    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bddf636-11b7-4b2c-864e-0dee5d23ce04-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.699952 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.699909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28352c09-054f-49cd-9d31-91c528433d45" (UID: "28352c09-054f-49cd-9d31-91c528433d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:43.700233 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.700184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" (UID: "4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:43.701550 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.701526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl" (OuterVolumeSpecName: "kube-api-access-m5qxl") pod "28352c09-054f-49cd-9d31-91c528433d45" (UID: "28352c09-054f-49cd-9d31-91c528433d45"). InnerVolumeSpecName "kube-api-access-m5qxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:43.701627 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.701579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2" (OuterVolumeSpecName: "kube-api-access-n2rv2") pod "3bddf636-11b7-4b2c-864e-0dee5d23ce04" (UID: "3bddf636-11b7-4b2c-864e-0dee5d23ce04"). InnerVolumeSpecName "kube-api-access-n2rv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:43.701955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.701940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3bddf636-11b7-4b2c-864e-0dee5d23ce04" (UID: "3bddf636-11b7-4b2c-864e-0dee5d23ce04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:43.701999 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.701972    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9" (OuterVolumeSpecName: "kube-api-access-wc8f9") pod "4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" (UID: "4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a"). InnerVolumeSpecName "kube-api-access-wc8f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800537    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qxl\" (UniqueName: \"kubernetes.io/projected/28352c09-054f-49cd-9d31-91c528433d45-kube-api-access-m5qxl\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800568    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2rv2\" (UniqueName: \"kubernetes.io/projected/3bddf636-11b7-4b2c-864e-0dee5d23ce04-kube-api-access-n2rv2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800577    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28352c09-054f-49cd-9d31-91c528433d45-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800586    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800595    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3bddf636-11b7-4b2c-864e-0dee5d23ce04-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:43.800626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:43.800605    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc8f9\" (UniqueName: \"kubernetes.io/projected/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a-kube-api-access-wc8f9\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:20:44.336408 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.336373    2580 generic.go:356] "Generic (PLEG): container finished" podID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" containerID="c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986" exitCode=0
Jul 27 21:20:44.336859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.336423    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"
Jul 27 21:20:44.336859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.336461    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" event={"ID":"3bddf636-11b7-4b2c-864e-0dee5d23ce04","Type":"ContainerDied","Data":"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"}
Jul 27 21:20:44.336859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.336514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs" event={"ID":"3bddf636-11b7-4b2c-864e-0dee5d23ce04","Type":"ContainerDied","Data":"49747bc9d53c05ca63215d6b8706ab8bac74dfd986b1222d66416110abfdb628"}
Jul 27 21:20:44.336859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.336548    2580 scope.go:117] "RemoveContainer" containerID="c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"
Jul 27 21:20:44.337972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.337778    2580 generic.go:356] "Generic (PLEG): container finished" podID="28352c09-054f-49cd-9d31-91c528433d45" containerID="2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61" exitCode=0
Jul 27 21:20:44.337972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.337824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"
Jul 27 21:20:44.337972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.337865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" event={"ID":"28352c09-054f-49cd-9d31-91c528433d45","Type":"ContainerDied","Data":"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"}
Jul 27 21:20:44.337972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.337898    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8" event={"ID":"28352c09-054f-49cd-9d31-91c528433d45","Type":"ContainerDied","Data":"5ee74536d154de2a0a1f4beb04f11567024dfa38f41795946da994bb3125c80c"}
Jul 27 21:20:44.339357 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.339336    2580 generic.go:356] "Generic (PLEG): container finished" podID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerID="4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631" exitCode=0
Jul 27 21:20:44.339473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.339367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" event={"ID":"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a","Type":"ContainerDied","Data":"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"}
Jul 27 21:20:44.339473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.339383    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"
Jul 27 21:20:44.339473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.339414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz" event={"ID":"4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a","Type":"ContainerDied","Data":"ea30a1e6c349adcfc30ae8a6fd76a68a27541cc7b777b9c4f930b51d559b26f7"}
Jul 27 21:20:44.347640 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.347618    2580 scope.go:117] "RemoveContainer" containerID="c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"
Jul 27 21:20:44.347930 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:20:44.347913    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986\": container with ID starting with c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986 not found: ID does not exist" containerID="c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"
Jul 27 21:20:44.348010 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.347940    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986"} err="failed to get container status \"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986\": rpc error: code = NotFound desc = could not find container \"c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986\": container with ID starting with c8cbe634742d499324791c976a365d5210a644e29bdd0994f89b46f9a359d986 not found: ID does not exist"
Jul 27 21:20:44.348010 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.347957    2580 scope.go:117] "RemoveContainer" containerID="2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"
Jul 27 21:20:44.356183 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.356167    2580 scope.go:117] "RemoveContainer" containerID="2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"
Jul 27 21:20:44.356473 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:20:44.356456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61\": container with ID starting with 2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61 not found: ID does not exist" containerID="2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"
Jul 27 21:20:44.356550 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.356486    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61"} err="failed to get container status \"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61\": rpc error: code = NotFound desc = could not find container \"2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61\": container with ID starting with 2be60cdde0ff2ee05d985272ba10aaa7c8680b3f531d8397c70218a2f3727f61 not found: ID does not exist"
Jul 27 21:20:44.356550 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.356513    2580 scope.go:117] "RemoveContainer" containerID="4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"
Jul 27 21:20:44.365772 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.365752    2580 scope.go:117] "RemoveContainer" containerID="4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"
Jul 27 21:20:44.366006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.365986    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:44.366006 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:20:44.365999    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631\": container with ID starting with 4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631 not found: ID does not exist" containerID="4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"
Jul 27 21:20:44.366095 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.366021    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631"} err="failed to get container status \"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631\": rpc error: code = NotFound desc = could not find container \"4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631\": container with ID starting with 4b72064d387d457633b6a0ba1658be96224b9b47717fbf845255a8e8fd2b7631 not found: ID does not exist"
Jul 27 21:20:44.369502 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.369474    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-338bbf37-7f485dc8b7-8ssvs"]
Jul 27 21:20:44.379765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.379732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:44.382581 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.382558    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-338bbf37-6788b9d676-t8jzz"]
Jul 27 21:20:44.392378 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.392352    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:44.396024 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.395994    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-338bbf37-58df449d97-c9dd8"]
Jul 27 21:20:44.805265 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.805231    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28352c09-054f-49cd-9d31-91c528433d45" path="/var/lib/kubelet/pods/28352c09-054f-49cd-9d31-91c528433d45/volumes"
Jul 27 21:20:44.805631 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.805618    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" path="/var/lib/kubelet/pods/3bddf636-11b7-4b2c-864e-0dee5d23ce04/volumes"
Jul 27 21:20:44.805920 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:44.805909    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" path="/var/lib/kubelet/pods/4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a/volumes"
Jul 27 21:20:46.109659 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:46.109624    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:46.199688 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:46.199661    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:47.934534 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:47.934494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:48.089576 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:48.089539    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:48.340606 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:48.340563    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:48.600744 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:48.600653    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:48.940792 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:48.940694    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:49.674933 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:49.674902    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:50.992840 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:50.992799    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.073227 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073186    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:20:51.073599 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerName="payload-pre-processing"
Jul 27 21:20:51.073648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073601    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerName="payload-pre-processing"
Jul 27 21:20:51.073648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28352c09-054f-49cd-9d31-91c528433d45" containerName="payload-processing"
Jul 27 21:20:51.073648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073624    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28352c09-054f-49cd-9d31-91c528433d45" containerName="payload-processing"
Jul 27 21:20:51.073648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073630    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" containerName="maas-api"
Jul 27 21:20:51.073648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073636    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" containerName="maas-api"
Jul 27 21:20:51.073809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073687    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bddf636-11b7-4b2c-864e-0dee5d23ce04" containerName="maas-api"
Jul 27 21:20:51.073809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073695    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec69edc-22ef-45bb-9ba6-f9fb1ddd0e8a" containerName="payload-pre-processing"
Jul 27 21:20:51.073809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.073704    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28352c09-054f-49cd-9d31-91c528433d45" containerName="payload-processing"
Jul 27 21:20:51.076290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.076256    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.078562 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.078545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.080889 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.080874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-54ece799-serving-cert\""
Jul 27 21:20:51.085929 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.085902    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:20:51.147699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.147644    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.164240 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.164178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.164399 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.164274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.164399 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.164297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mdk\" (UniqueName: \"kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.247562 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.247461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.265684 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.265647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.265842 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.265705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.265842 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.265724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mdk\" (UniqueName: \"kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.266488 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.266465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.268334 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.268313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.282561 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.282533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mdk\" (UniqueName: \"kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk\") pod \"maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.390620 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.390581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:51.536992 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.536907    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:20:51.541962 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:20:51.541932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613be516_5c19_462b_8aa0_700ff64a8972.slice/crio-880491d53b372312f49dea6bb1fa7dff3f83ca6821518f5c4b5b4e03d5165261 WatchSource:0}: Error finding container 880491d53b372312f49dea6bb1fa7dff3f83ca6821518f5c4b5b4e03d5165261: Status 404 returned error can't find the container with id 880491d53b372312f49dea6bb1fa7dff3f83ca6821518f5c4b5b4e03d5165261
Jul 27 21:20:51.695491 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.695459    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.794619 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.794510    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.799370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.799343    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:20:51.804090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.804069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.808057 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.808033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-54ece799\""
Jul 27 21:20:51.822823 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.822788    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:20:51.871077 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.871030    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.871304 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.871135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qh5\" (UniqueName: \"kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.971824 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.971791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qh5\" (UniqueName: \"kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.971974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.971884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.972585 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.972556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:51.981385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:51.981358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qh5\" (UniqueName: \"kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5\") pod \"payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:52.115079 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.114980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:52.265015 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.264985    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:20:52.267497 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:20:52.267469    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cac18c_0546_49e1_b5a0_3fc6e9b59ad4.slice/crio-a7265874ebcab883d95141be596e33bce038c732aa0bb88b2191a57cd80d60dd WatchSource:0}: Error finding container a7265874ebcab883d95141be596e33bce038c732aa0bb88b2191a57cd80d60dd: Status 404 returned error can't find the container with id a7265874ebcab883d95141be596e33bce038c732aa0bb88b2191a57cd80d60dd
Jul 27 21:20:52.382773 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.382677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" event={"ID":"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4","Type":"ContainerStarted","Data":"77f996c302d5df45cd2056861afe8ca96edecf6b4ee45d232ad312f84ddb7a3a"}
Jul 27 21:20:52.382773 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.382728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" event={"ID":"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4","Type":"ContainerStarted","Data":"a7265874ebcab883d95141be596e33bce038c732aa0bb88b2191a57cd80d60dd"}
Jul 27 21:20:52.382773 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.382759    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:20:52.384361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.384335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" event={"ID":"613be516-5c19-462b-8aa0-700ff64a8972","Type":"ContainerStarted","Data":"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"}
Jul 27 21:20:52.384504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.384383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" event={"ID":"613be516-5c19-462b-8aa0-700ff64a8972","Type":"ContainerStarted","Data":"880491d53b372312f49dea6bb1fa7dff3f83ca6821518f5c4b5b4e03d5165261"}
Jul 27 21:20:52.384504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.384409    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:20:52.397700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.397645    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" podStartSLOduration=1.397628526 podStartE2EDuration="1.397628526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:52.396843943 +0000 UTC m=+1758.170776364" watchObservedRunningTime="2026-07-27 21:20:52.397628526 +0000 UTC m=+1758.171560997"
Jul 27 21:20:52.417910 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:52.417855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" podStartSLOduration=1.021922646 podStartE2EDuration="1.417835086s" totalImagesPullingTime="395.91244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:52.415338241 +0000 UTC m=+1758.189270661" watchObservedRunningTime="2026-07-27 21:20:52.417835086 +0000 UTC m=+1758.191767505"
Jul 27 21:20:58.395021 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:20:58.394993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:21:03.391608 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:03.391575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:21:27.240349 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.240304    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:21:27.240723 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.240581    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" podUID="613be516-5c19-462b-8aa0-700ff64a8972" containerName="maas-api" containerID="cri-o://4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e" gracePeriod=30
Jul 27 21:21:27.265028 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.264993    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:27.304794 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.295783    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:21:27.304794 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.296108    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" podUID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" containerName="payload-pre-processing" containerID="cri-o://77f996c302d5df45cd2056861afe8ca96edecf6b4ee45d232ad312f84ddb7a3a" gracePeriod=30
Jul 27 21:21:27.365233 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.365175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:27.466638 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.466568    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:27.557876 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.557850    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:21:27.571250 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.571199    2580 generic.go:356] "Generic (PLEG): container finished" podID="613be516-5c19-462b-8aa0-700ff64a8972" containerID="4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e" exitCode=0
Jul 27 21:21:27.571452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.571269    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"
Jul 27 21:21:27.571452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.571276    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" event={"ID":"613be516-5c19-462b-8aa0-700ff64a8972","Type":"ContainerDied","Data":"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"}
Jul 27 21:21:27.571452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.571369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62" event={"ID":"613be516-5c19-462b-8aa0-700ff64a8972","Type":"ContainerDied","Data":"880491d53b372312f49dea6bb1fa7dff3f83ca6821518f5c4b5b4e03d5165261"}
Jul 27 21:21:27.571452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.571415    2580 scope.go:117] "RemoveContainer" containerID="4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"
Jul 27 21:21:27.573193 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.573166    2580 generic.go:356] "Generic (PLEG): container finished" podID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" containerID="77f996c302d5df45cd2056861afe8ca96edecf6b4ee45d232ad312f84ddb7a3a" exitCode=0
Jul 27 21:21:27.573396 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.573195    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" event={"ID":"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4","Type":"ContainerDied","Data":"77f996c302d5df45cd2056861afe8ca96edecf6b4ee45d232ad312f84ddb7a3a"}
Jul 27 21:21:27.590955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.590921    2580 scope.go:117] "RemoveContainer" containerID="4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"
Jul 27 21:21:27.591264 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:21:27.591238    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e\": container with ID starting with 4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e not found: ID does not exist" containerID="4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"
Jul 27 21:21:27.591383 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.591272    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e"} err="failed to get container status \"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e\": rpc error: code = NotFound desc = could not find container \"4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e\": container with ID starting with 4abbd50ac1876ec100d79866983fd4bb68c0807bf6dfe6be7b099a914076e67e not found: ID does not exist"
Jul 27 21:21:27.592927 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.592902    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:21:27.615272 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615239    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67qh5\" (UniqueName: \"kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5\") pod \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") "
Jul 27 21:21:27.615451 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca\") pod \"613be516-5c19-462b-8aa0-700ff64a8972\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") "
Jul 27 21:21:27.615451 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume\") pod \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\" (UID: \"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4\") "
Jul 27 21:21:27.615451 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls\") pod \"613be516-5c19-462b-8aa0-700ff64a8972\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") "
Jul 27 21:21:27.615451 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94mdk\" (UniqueName: \"kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk\") pod \"613be516-5c19-462b-8aa0-700ff64a8972\" (UID: \"613be516-5c19-462b-8aa0-700ff64a8972\") "
Jul 27 21:21:27.615683 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.615661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "613be516-5c19-462b-8aa0-700ff64a8972" (UID: "613be516-5c19-462b-8aa0-700ff64a8972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:21:27.616061 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.616031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" (UID: "91cac18c-0546-49e1-b5a0-3fc6e9b59ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:21:27.617753 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.617719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5" (OuterVolumeSpecName: "kube-api-access-67qh5") pod "91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" (UID: "91cac18c-0546-49e1-b5a0-3fc6e9b59ad4"). InnerVolumeSpecName "kube-api-access-67qh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:27.617851 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.617774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "613be516-5c19-462b-8aa0-700ff64a8972" (UID: "613be516-5c19-462b-8aa0-700ff64a8972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:21:27.617851 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.617781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk" (OuterVolumeSpecName: "kube-api-access-94mdk") pod "613be516-5c19-462b-8aa0-700ff64a8972" (UID: "613be516-5c19-462b-8aa0-700ff64a8972"). InnerVolumeSpecName "kube-api-access-94mdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:27.716894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.716849    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/613be516-5c19-462b-8aa0-700ff64a8972-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:21:27.716894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.716885    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mdk\" (UniqueName: \"kubernetes.io/projected/613be516-5c19-462b-8aa0-700ff64a8972-kube-api-access-94mdk\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:21:27.716894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.716899    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qh5\" (UniqueName: \"kubernetes.io/projected/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-kube-api-access-67qh5\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:21:27.717154 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.716912    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/613be516-5c19-462b-8aa0-700ff64a8972-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:21:27.717154 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.716924    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:21:27.892280 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.892239    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:21:27.896109 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:27.896079    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54ece799-6fd68c9b5c-j5x62"]
Jul 27 21:21:28.580336 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.580242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw" event={"ID":"91cac18c-0546-49e1-b5a0-3fc6e9b59ad4","Type":"ContainerDied","Data":"a7265874ebcab883d95141be596e33bce038c732aa0bb88b2191a57cd80d60dd"}
Jul 27 21:21:28.580336 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.580267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"
Jul 27 21:21:28.580336 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.580305    2580 scope.go:117] "RemoveContainer" containerID="77f996c302d5df45cd2056861afe8ca96edecf6b4ee45d232ad312f84ddb7a3a"
Jul 27 21:21:28.601020 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.600992    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:21:28.604885 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.604856    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54ece799-56bfbcc85b-wqwkw"]
Jul 27 21:21:28.805183 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.805146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613be516-5c19-462b-8aa0-700ff64a8972" path="/var/lib/kubelet/pods/613be516-5c19-462b-8aa0-700ff64a8972/volumes"
Jul 27 21:21:28.805564 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:28.805549    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" path="/var/lib/kubelet/pods/91cac18c-0546-49e1-b5a0-3fc6e9b59ad4/volumes"
Jul 27 21:21:31.919132 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:31.919102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:33.150010 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:33.149971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:33.379803 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:33.379759    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:33.621900 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:33.621841    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:33.912423 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:33.912330    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:34.133785 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:34.133733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:35.691894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:35.691865    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:21:35.696300 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:35.696276    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:21:36.184185 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.184143    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:36.267292 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267259    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:21:36.267738 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" containerName="payload-pre-processing"
Jul 27 21:21:36.267849 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267741    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" containerName="payload-pre-processing"
Jul 27 21:21:36.267849 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267761    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613be516-5c19-462b-8aa0-700ff64a8972" containerName="maas-api"
Jul 27 21:21:36.267849 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="613be516-5c19-462b-8aa0-700ff64a8972" containerName="maas-api"
Jul 27 21:21:36.268003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267863    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cac18c-0546-49e1-b5a0-3fc6e9b59ad4" containerName="payload-pre-processing"
Jul 27 21:21:36.268003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.267882    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="613be516-5c19-462b-8aa0-700ff64a8972" containerName="maas-api"
Jul 27 21:21:36.271923 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.271892    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:36.280131 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.280108    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.283332 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.283020    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f154b44a-serving-cert\""
Jul 27 21:21:36.283332 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.283067    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:21:36.396148 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.396103    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7dd\" (UniqueName: \"kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.396385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.396193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.396385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.396275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.423182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.423129    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:36.497505 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.497461    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.497730 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.497524    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.497730 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.497563    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7dd\" (UniqueName: \"kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.497730 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:21:36.497658    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-serving-cert: secret "maas-api-e2e-derive-f154b44a-serving-cert" not found
Jul 27 21:21:36.497904 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:21:36.497743    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls podName:adce8b86-8286-4620-8047-8cd7a2f95665 nodeName:}" failed. No retries permitted until 2026-07-27 21:21:36.997720092 +0000 UTC m=+1802.771652493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls") pod "maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" (UID: "adce8b86-8286-4620-8047-8cd7a2f95665") : secret "maas-api-e2e-derive-f154b44a-serving-cert" not found
Jul 27 21:21:36.498328 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.498311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.506330 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.506303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7dd\" (UniqueName: \"kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:36.835657 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.835540    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:36.935792 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.935752    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:36.979467 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.979429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:21:36.983386 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.983362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:36.986375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.986348    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f154b44a-dockercfg-w7hhg\""
Jul 27 21:21:36.986885 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.986862    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f154b44a\""
Jul 27 21:21:36.996000 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:36.995962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:21:37.002725 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.002693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:37.002936 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:21:37.002864    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-serving-cert: secret "maas-api-e2e-derive-f154b44a-serving-cert" not found
Jul 27 21:21:37.003005 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:21:37.002953    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls podName:adce8b86-8286-4620-8047-8cd7a2f95665 nodeName:}" failed. No retries permitted until 2026-07-27 21:21:38.002930464 +0000 UTC m=+1803.776862876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls") pod "maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" (UID: "adce8b86-8286-4620-8047-8cd7a2f95665") : secret "maas-api-e2e-derive-f154b44a-serving-cert" not found
Jul 27 21:21:37.103479 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.103371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppts\" (UniqueName: \"kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.103479 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.103424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.204402 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.204349    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppts\" (UniqueName: \"kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.204402 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.204404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.205098 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.205078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.213425 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.213397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppts\" (UniqueName: \"kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts\") pod \"payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") " pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.296955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.296917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.308950 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.308915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:37.448708 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.448681    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:21:37.450775 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:21:37.450740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744545d5_17bd_4aca_a7ba_7c19d6d6ebd6.slice/crio-f2b729fe9cde3ed77343432e6f1ef376d9de4c98d864569cf1cb676a5537760a WatchSource:0}: Error finding container f2b729fe9cde3ed77343432e6f1ef376d9de4c98d864569cf1cb676a5537760a: Status 404 returned error can't find the container with id f2b729fe9cde3ed77343432e6f1ef376d9de4c98d864569cf1cb676a5537760a
Jul 27 21:21:37.627989 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.627886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" event={"ID":"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6","Type":"ContainerStarted","Data":"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"}
Jul 27 21:21:37.627989 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.627946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" event={"ID":"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6","Type":"ContainerStarted","Data":"f2b729fe9cde3ed77343432e6f1ef376d9de4c98d864569cf1cb676a5537760a"}
Jul 27 21:21:37.628242 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.627997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:21:37.662654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:37.662602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" podStartSLOduration=1.66258352 podStartE2EDuration="1.66258352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:21:37.660315068 +0000 UTC m=+1803.434247487" watchObservedRunningTime="2026-07-27 21:21:37.66258352 +0000 UTC m=+1803.436515939"
Jul 27 21:21:38.013784 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:38.013739    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:38.016742 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:38.016710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:38.094670 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:38.094627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:38.244675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:38.244646    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:21:38.246810 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:21:38.246777    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadce8b86_8286_4620_8047_8cd7a2f95665.slice/crio-648ddcbf0de06d81908587d7633a1dfa13d0519efa7af4438c4c6a248210024a WatchSource:0}: Error finding container 648ddcbf0de06d81908587d7633a1dfa13d0519efa7af4438c4c6a248210024a: Status 404 returned error can't find the container with id 648ddcbf0de06d81908587d7633a1dfa13d0519efa7af4438c4c6a248210024a
Jul 27 21:21:38.634693 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:38.634649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" event={"ID":"adce8b86-8286-4620-8047-8cd7a2f95665","Type":"ContainerStarted","Data":"648ddcbf0de06d81908587d7633a1dfa13d0519efa7af4438c4c6a248210024a"}
Jul 27 21:21:39.641783 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:39.641747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" event={"ID":"adce8b86-8286-4620-8047-8cd7a2f95665","Type":"ContainerStarted","Data":"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"}
Jul 27 21:21:39.642206 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:39.641849    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:39.665137 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:39.665078    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" podStartSLOduration=3.31782952 podStartE2EDuration="3.665062756s" totalImagesPullingTime="347.233236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:21:39.662921864 +0000 UTC m=+1805.436854284" watchObservedRunningTime="2026-07-27 21:21:39.665062756 +0000 UTC m=+1805.438995175"
Jul 27 21:21:45.652650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:45.652616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:21:48.637415 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:21:48.637379    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:22:01.192881 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.192843    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:22:01.193709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.193646    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" podUID="adce8b86-8286-4620-8047-8cd7a2f95665" containerName="maas-api" containerID="cri-o://14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759" gracePeriod=30
Jul 27 21:22:01.220690 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.220654    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:01.252150 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.252107    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:22:01.252624 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.252351    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" podUID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" containerName="payload-processing" containerID="cri-o://79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5" gracePeriod=30
Jul 27 21:22:01.321760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.321728    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:01.422834 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.422749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:01.507539 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.507511    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:22:01.534778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.534751    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:22:01.639007 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.638975    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") pod \"adce8b86-8286-4620-8047-8cd7a2f95665\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") "
Jul 27 21:22:01.639243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kppts\" (UniqueName: \"kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts\") pod \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") "
Jul 27 21:22:01.639243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639054    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume\") pod \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\" (UID: \"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6\") "
Jul 27 21:22:01.639243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca\") pod \"adce8b86-8286-4620-8047-8cd7a2f95665\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") "
Jul 27 21:22:01.639243 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639200    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz7dd\" (UniqueName: \"kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd\") pod \"adce8b86-8286-4620-8047-8cd7a2f95665\" (UID: \"adce8b86-8286-4620-8047-8cd7a2f95665\") "
Jul 27 21:22:01.639565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639528    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adce8b86-8286-4620-8047-8cd7a2f95665" (UID: "adce8b86-8286-4620-8047-8cd7a2f95665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:01.639687 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.639538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" (UID: "744545d5-17bd-4aca-a7ba-7c19d6d6ebd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:01.641359 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.641332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adce8b86-8286-4620-8047-8cd7a2f95665" (UID: "adce8b86-8286-4620-8047-8cd7a2f95665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:01.641446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.641361    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd" (OuterVolumeSpecName: "kube-api-access-wz7dd") pod "adce8b86-8286-4620-8047-8cd7a2f95665" (UID: "adce8b86-8286-4620-8047-8cd7a2f95665"). InnerVolumeSpecName "kube-api-access-wz7dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:01.641446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.641406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts" (OuterVolumeSpecName: "kube-api-access-kppts") pod "744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" (UID: "744545d5-17bd-4aca-a7ba-7c19d6d6ebd6"). InnerVolumeSpecName "kube-api-access-kppts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:01.740543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.740454    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adce8b86-8286-4620-8047-8cd7a2f95665-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:22:01.740543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.740482    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7dd\" (UniqueName: \"kubernetes.io/projected/adce8b86-8286-4620-8047-8cd7a2f95665-kube-api-access-wz7dd\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:22:01.740543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.740493    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adce8b86-8286-4620-8047-8cd7a2f95665-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:22:01.740543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.740502    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppts\" (UniqueName: \"kubernetes.io/projected/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-kube-api-access-kppts\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:22:01.740543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.740512    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:22:01.756325 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.756287    2580 generic.go:356] "Generic (PLEG): container finished" podID="adce8b86-8286-4620-8047-8cd7a2f95665" containerID="14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759" exitCode=0
Jul 27 21:22:01.756486 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.756329    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"
Jul 27 21:22:01.756486 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.756366    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" event={"ID":"adce8b86-8286-4620-8047-8cd7a2f95665","Type":"ContainerDied","Data":"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"}
Jul 27 21:22:01.756486 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.756420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8" event={"ID":"adce8b86-8286-4620-8047-8cd7a2f95665","Type":"ContainerDied","Data":"648ddcbf0de06d81908587d7633a1dfa13d0519efa7af4438c4c6a248210024a"}
Jul 27 21:22:01.756486 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.756443    2580 scope.go:117] "RemoveContainer" containerID="14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"
Jul 27 21:22:01.757846 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.757821    2580 generic.go:356] "Generic (PLEG): container finished" podID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" containerID="79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5" exitCode=0
Jul 27 21:22:01.757962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.757860    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"
Jul 27 21:22:01.757962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.757894    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" event={"ID":"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6","Type":"ContainerDied","Data":"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"}
Jul 27 21:22:01.757962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.757950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2" event={"ID":"744545d5-17bd-4aca-a7ba-7c19d6d6ebd6","Type":"ContainerDied","Data":"f2b729fe9cde3ed77343432e6f1ef376d9de4c98d864569cf1cb676a5537760a"}
Jul 27 21:22:01.766592 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.766572    2580 scope.go:117] "RemoveContainer" containerID="14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"
Jul 27 21:22:01.766901 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:22:01.766882    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759\": container with ID starting with 14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759 not found: ID does not exist" containerID="14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"
Jul 27 21:22:01.766970 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.766910    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759"} err="failed to get container status \"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759\": rpc error: code = NotFound desc = could not find container \"14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759\": container with ID starting with 14df3c450f6ee8cb55bd955f1111cbf9b5ebb1e890e31fa979373c8fd5202759 not found: ID does not exist"
Jul 27 21:22:01.766970 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.766928    2580 scope.go:117] "RemoveContainer" containerID="79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"
Jul 27 21:22:01.775714 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.775696    2580 scope.go:117] "RemoveContainer" containerID="79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"
Jul 27 21:22:01.776009 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:22:01.775984    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5\": container with ID starting with 79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5 not found: ID does not exist" containerID="79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"
Jul 27 21:22:01.776072 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.776019    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5"} err="failed to get container status \"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5\": rpc error: code = NotFound desc = could not find container \"79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5\": container with ID starting with 79b9eb7d4826eeee0c9f3034d637546e4cd2b518efc7eb9c41c9b98046b842a5 not found: ID does not exist"
Jul 27 21:22:01.781796 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.781771    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:22:01.785059 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.785037    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f154b44a-6b84bf4d46-bmqm2"]
Jul 27 21:22:01.794474 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.794447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:22:01.798432 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:01.798407    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f154b44a-5455c9fd98-p5lr8"]
Jul 27 21:22:02.804512 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:02.804476    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" path="/var/lib/kubelet/pods/744545d5-17bd-4aca-a7ba-7c19d6d6ebd6/volumes"
Jul 27 21:22:02.804963 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:02.804875    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adce8b86-8286-4620-8047-8cd7a2f95665" path="/var/lib/kubelet/pods/adce8b86-8286-4620-8047-8cd7a2f95665/volumes"
Jul 27 21:22:05.873888 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:05.873847    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:06.036298 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:06.034046    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:06.279390 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:06.279354    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:06.773675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:06.773637    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:06.863562 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:06.863517    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:06.963615 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:06.963573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:08.468038 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:08.468006    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:14.746325 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:14.746187    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:14.876456 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:14.876419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:15.002775 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:15.002672    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:15.098019 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:15.097971    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:15.503624 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:15.503584    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:41.525226 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:41.525185    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:47.539749 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:47.539700    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:47.649869 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:47.649809    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:47.743974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:22:47.743943    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:03.645113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:03.645027    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:09.506647 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:09.506612    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:09.610257 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:09.610193    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:09.706758 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:09.706719    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:20.249965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:20.249925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:21.067132 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:21.067098    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:27.066474 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:27.066436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:27.226848 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:27.226804    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:27.363726 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:27.363633    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:27.451368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:27.451336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:28.212178 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:28.212134    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:28.309760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:28.309709    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:30.021154 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:30.021119    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:36.108848 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:36.108806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:36.209642 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:36.209601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:36.841374 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:36.841329    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:42.838716 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:42.838679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:42.940443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:42.940404    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:43.073399 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:43.073357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:43.481305 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:43.481270    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:43.812406 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:43.812316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:46.361252 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.361197    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:46.455074 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.455032    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:46.460175 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460145    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:23:46.460774 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460751    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" containerName="payload-processing"
Jul 27 21:23:46.460831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460780    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" containerName="payload-processing"
Jul 27 21:23:46.460873 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460835    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adce8b86-8286-4620-8047-8cd7a2f95665" containerName="maas-api"
Jul 27 21:23:46.460873 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adce8b86-8286-4620-8047-8cd7a2f95665" containerName="maas-api"
Jul 27 21:23:46.460956 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460934    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="744545d5-17bd-4aca-a7ba-7c19d6d6ebd6" containerName="payload-processing"
Jul 27 21:23:46.460956 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.460948    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adce8b86-8286-4620-8047-8cd7a2f95665" containerName="maas-api"
Jul 27 21:23:46.464298 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.464278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.467277 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.467256    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c5876a31-serving-cert\""
Jul 27 21:23:46.480431 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.480408    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:23:46.496668 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.496628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqg7r\" (UniqueName: \"kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.496822 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.496687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.496822 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.496745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.532127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.532087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:46.597613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.597571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.597613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.597619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.597826 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.597680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqg7r\" (UniqueName: \"kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.597826 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:23:46.597799    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-serving-cert: secret "maas-api-e2e-mt-c5876a31-serving-cert" not found
Jul 27 21:23:46.597899 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:23:46.597869    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls podName:fd51454f-202b-4256-a49c-85366e57ac6c nodeName:}" failed. No retries permitted until 2026-07-27 21:23:47.097848546 +0000 UTC m=+1932.871780943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls") pod "maas-api-e2e-mt-c5876a31-578bb89c69-fg424" (UID: "fd51454f-202b-4256-a49c-85366e57ac6c") : secret "maas-api-e2e-mt-c5876a31-serving-cert" not found
Jul 27 21:23:46.598286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.598269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:46.606377 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:46.606361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqg7r\" (UniqueName: \"kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:47.011286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.011175    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:47.103129 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.103072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:47.103381 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:23:47.103273    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-serving-cert: secret "maas-api-e2e-mt-c5876a31-serving-cert" not found
Jul 27 21:23:47.103381 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:23:47.103348    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls podName:fd51454f-202b-4256-a49c-85366e57ac6c nodeName:}" failed. No retries permitted until 2026-07-27 21:23:48.10333365 +0000 UTC m=+1933.877266048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls") pod "maas-api-e2e-mt-c5876a31-578bb89c69-fg424" (UID: "fd51454f-202b-4256-a49c-85366e57ac6c") : secret "maas-api-e2e-mt-c5876a31-serving-cert" not found
Jul 27 21:23:47.110972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.110924    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:47.119453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.119429    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:23:47.124057 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.124023    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.127312 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.127290    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c5876a31\""
Jul 27 21:23:47.134960 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.134933    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:23:47.157380 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.157349    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:23:47.160791 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.160775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.163222 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.163191    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c5876a31-dockercfg-kt4x9\""
Jul 27 21:23:47.173176 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.173146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:23:47.204040 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.203979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvllr\" (UniqueName: \"kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.204260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.204115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sf5s\" (UniqueName: \"kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.204260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.204178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.204260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.204238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.220602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.220566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:47.305035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.304874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvllr\" (UniqueName: \"kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.305035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.304929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sf5s\" (UniqueName: \"kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.305035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.304966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.305035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.304985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.314090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.305665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.314090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.305710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.314090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.313897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvllr\" (UniqueName: \"kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr\") pod \"payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.314231 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.314118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sf5s\" (UniqueName: \"kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s\") pod \"payload-processing-e2e-mt-c5876a31-7845485958-fzkvx\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") " pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.437589 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.437550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:47.472467 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.472426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:47.601941 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.601764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:23:47.605807 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:23:47.605779    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550ef780_5b08_4d03_9ca8_0c120a98be77.slice/crio-cf52938135e12f4595601062c3758474d2c4e1c242b542da24f1411d9aa13c84 WatchSource:0}: Error finding container cf52938135e12f4595601062c3758474d2c4e1c242b542da24f1411d9aa13c84: Status 404 returned error can't find the container with id cf52938135e12f4595601062c3758474d2c4e1c242b542da24f1411d9aa13c84
Jul 27 21:23:47.651542 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:47.651518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:23:47.653862 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:23:47.653831    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78510440_9d87_4934_ab79_e56e4b508bb0.slice/crio-2505c914d3cb9ac27889c2cbc93d91e63569ad6441e2f7417dd5ede69c937e41 WatchSource:0}: Error finding container 2505c914d3cb9ac27889c2cbc93d91e63569ad6441e2f7417dd5ede69c937e41: Status 404 returned error can't find the container with id 2505c914d3cb9ac27889c2cbc93d91e63569ad6441e2f7417dd5ede69c937e41
Jul 27 21:23:48.113034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.112993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:48.115755 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.115726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"maas-api-e2e-mt-c5876a31-578bb89c69-fg424\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:48.265931 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.265888    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" event={"ID":"550ef780-5b08-4d03-9ca8-0c120a98be77","Type":"ContainerStarted","Data":"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"}
Jul 27 21:23:48.266151 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.265957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" event={"ID":"550ef780-5b08-4d03-9ca8-0c120a98be77","Type":"ContainerStarted","Data":"cf52938135e12f4595601062c3758474d2c4e1c242b542da24f1411d9aa13c84"}
Jul 27 21:23:48.266151 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.265984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:23:48.267709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.267679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" event={"ID":"78510440-9d87-4934-ab79-e56e4b508bb0","Type":"ContainerStarted","Data":"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"}
Jul 27 21:23:48.267951 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.267732    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" event={"ID":"78510440-9d87-4934-ab79-e56e4b508bb0","Type":"ContainerStarted","Data":"2505c914d3cb9ac27889c2cbc93d91e63569ad6441e2f7417dd5ede69c937e41"}
Jul 27 21:23:48.267951 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.267762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:48.273880 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.273857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:48.284462 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.284408    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" podStartSLOduration=1.284389011 podStartE2EDuration="1.284389011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:48.284248248 +0000 UTC m=+1934.058180668" watchObservedRunningTime="2026-07-27 21:23:48.284389011 +0000 UTC m=+1934.058321432"
Jul 27 21:23:48.301946 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.301896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" podStartSLOduration=1.301876768 podStartE2EDuration="1.301876768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:48.301277929 +0000 UTC m=+1934.075210352" watchObservedRunningTime="2026-07-27 21:23:48.301876768 +0000 UTC m=+1934.075809231"
Jul 27 21:23:48.419700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:48.419673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:23:48.421907 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:23:48.421882    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd51454f_202b_4256_a49c_85366e57ac6c.slice/crio-01ea851b44bc254aa5dc946c4a43f3dbcb3b3890adb5dccf0179c9d9d7472bac WatchSource:0}: Error finding container 01ea851b44bc254aa5dc946c4a43f3dbcb3b3890adb5dccf0179c9d9d7472bac: Status 404 returned error can't find the container with id 01ea851b44bc254aa5dc946c4a43f3dbcb3b3890adb5dccf0179c9d9d7472bac
Jul 27 21:23:49.274302 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:49.274261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" event={"ID":"fd51454f-202b-4256-a49c-85366e57ac6c","Type":"ContainerStarted","Data":"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"}
Jul 27 21:23:49.274765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:49.274330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" event={"ID":"fd51454f-202b-4256-a49c-85366e57ac6c","Type":"ContainerStarted","Data":"01ea851b44bc254aa5dc946c4a43f3dbcb3b3890adb5dccf0179c9d9d7472bac"}
Jul 27 21:23:49.274765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:49.274508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:49.290724 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:49.290665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" podStartSLOduration=2.824229241 podStartE2EDuration="3.290644821s" totalImagesPullingTime="466.41558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:49.289096618 +0000 UTC m=+1935.063029040" watchObservedRunningTime="2026-07-27 21:23:49.290644821 +0000 UTC m=+1935.064577243"
Jul 27 21:23:50.882610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:50.882575    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:52.084731 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:52.084693    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:55.287453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:55.287421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:23:57.948777 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:57.948723    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:58.049378 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:58.049325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:58.987607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:58.987573    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:59.042367 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:59.042333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:59.277380 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:59.277285    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:23:59.277380 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:23:59.277359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:24:08.394333 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.394298    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:24:08.394747 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.394581    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" podUID="fd51454f-202b-4256-a49c-85366e57ac6c" containerName="maas-api" containerID="cri-o://a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf" gracePeriod=30
Jul 27 21:24:08.419440 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.419406    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:08.445267 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.445204    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:24:08.445456 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.445429    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" podUID="78510440-9d87-4934-ab79-e56e4b508bb0" containerName="payload-processing" containerID="cri-o://50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77" gracePeriod=30
Jul 27 21:24:08.453295 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.453258    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:24:08.453659 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.453627    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" podUID="550ef780-5b08-4d03-9ca8-0c120a98be77" containerName="payload-pre-processing" containerID="cri-o://1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02" gracePeriod=30
Jul 27 21:24:08.484386 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.484347    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:08.540013 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.539972    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:08.713034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.713005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:24:08.743133 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.743109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:24:08.749765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.749737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:24:08.807568 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807544    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") pod \"fd51454f-202b-4256-a49c-85366e57ac6c\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") "
Jul 27 21:24:08.807743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sf5s\" (UniqueName: \"kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s\") pod \"78510440-9d87-4934-ab79-e56e4b508bb0\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") "
Jul 27 21:24:08.807743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvllr\" (UniqueName: \"kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr\") pod \"550ef780-5b08-4d03-9ca8-0c120a98be77\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") "
Jul 27 21:24:08.807743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume\") pod \"550ef780-5b08-4d03-9ca8-0c120a98be77\" (UID: \"550ef780-5b08-4d03-9ca8-0c120a98be77\") "
Jul 27 21:24:08.807743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca\") pod \"fd51454f-202b-4256-a49c-85366e57ac6c\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") "
Jul 27 21:24:08.807962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqg7r\" (UniqueName: \"kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r\") pod \"fd51454f-202b-4256-a49c-85366e57ac6c\" (UID: \"fd51454f-202b-4256-a49c-85366e57ac6c\") "
Jul 27 21:24:08.807962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.807808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume\") pod \"78510440-9d87-4934-ab79-e56e4b508bb0\" (UID: \"78510440-9d87-4934-ab79-e56e4b508bb0\") "
Jul 27 21:24:08.808271 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.808199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd51454f-202b-4256-a49c-85366e57ac6c" (UID: "fd51454f-202b-4256-a49c-85366e57ac6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:08.808384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.808357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "550ef780-5b08-4d03-9ca8-0c120a98be77" (UID: "550ef780-5b08-4d03-9ca8-0c120a98be77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:08.808446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.808406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78510440-9d87-4934-ab79-e56e4b508bb0" (UID: "78510440-9d87-4934-ab79-e56e4b508bb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:08.810422 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.810384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r" (OuterVolumeSpecName: "kube-api-access-hqg7r") pod "fd51454f-202b-4256-a49c-85366e57ac6c" (UID: "fd51454f-202b-4256-a49c-85366e57ac6c"). InnerVolumeSpecName "kube-api-access-hqg7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:08.810422 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.810405    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr" (OuterVolumeSpecName: "kube-api-access-qvllr") pod "550ef780-5b08-4d03-9ca8-0c120a98be77" (UID: "550ef780-5b08-4d03-9ca8-0c120a98be77"). InnerVolumeSpecName "kube-api-access-qvllr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:08.810571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.810554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s" (OuterVolumeSpecName: "kube-api-access-7sf5s") pod "78510440-9d87-4934-ab79-e56e4b508bb0" (UID: "78510440-9d87-4934-ab79-e56e4b508bb0"). InnerVolumeSpecName "kube-api-access-7sf5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:08.810571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.810557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd51454f-202b-4256-a49c-85366e57ac6c" (UID: "fd51454f-202b-4256-a49c-85366e57ac6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:24:08.909533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909493    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd51454f-202b-4256-a49c-85366e57ac6c-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909525    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqg7r\" (UniqueName: \"kubernetes.io/projected/fd51454f-202b-4256-a49c-85366e57ac6c-kube-api-access-hqg7r\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909535    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78510440-9d87-4934-ab79-e56e4b508bb0-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909545    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd51454f-202b-4256-a49c-85366e57ac6c-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909854 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909555    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sf5s\" (UniqueName: \"kubernetes.io/projected/78510440-9d87-4934-ab79-e56e4b508bb0-kube-api-access-7sf5s\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909854 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909563    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvllr\" (UniqueName: \"kubernetes.io/projected/550ef780-5b08-4d03-9ca8-0c120a98be77-kube-api-access-qvllr\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:08.909854 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:08.909571    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ef780-5b08-4d03-9ca8-0c120a98be77-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:24:09.377164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.377128    2580 generic.go:356] "Generic (PLEG): container finished" podID="78510440-9d87-4934-ab79-e56e4b508bb0" containerID="50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77" exitCode=0
Jul 27 21:24:09.377369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.377171    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"
Jul 27 21:24:09.377369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.377237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" event={"ID":"78510440-9d87-4934-ab79-e56e4b508bb0","Type":"ContainerDied","Data":"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"}
Jul 27 21:24:09.377369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.377299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx" event={"ID":"78510440-9d87-4934-ab79-e56e4b508bb0","Type":"ContainerDied","Data":"2505c914d3cb9ac27889c2cbc93d91e63569ad6441e2f7417dd5ede69c937e41"}
Jul 27 21:24:09.377369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.377331    2580 scope.go:117] "RemoveContainer" containerID="50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"
Jul 27 21:24:09.378643 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.378625    2580 generic.go:356] "Generic (PLEG): container finished" podID="550ef780-5b08-4d03-9ca8-0c120a98be77" containerID="1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02" exitCode=0
Jul 27 21:24:09.378760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.378675    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"
Jul 27 21:24:09.378760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.378684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" event={"ID":"550ef780-5b08-4d03-9ca8-0c120a98be77","Type":"ContainerDied","Data":"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"}
Jul 27 21:24:09.378760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.378740    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx" event={"ID":"550ef780-5b08-4d03-9ca8-0c120a98be77","Type":"ContainerDied","Data":"cf52938135e12f4595601062c3758474d2c4e1c242b542da24f1411d9aa13c84"}
Jul 27 21:24:09.380106 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.380080    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd51454f-202b-4256-a49c-85366e57ac6c" containerID="a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf" exitCode=0
Jul 27 21:24:09.380253 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.380123    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"
Jul 27 21:24:09.380253 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.380126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" event={"ID":"fd51454f-202b-4256-a49c-85366e57ac6c","Type":"ContainerDied","Data":"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"}
Jul 27 21:24:09.380253 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.380245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424" event={"ID":"fd51454f-202b-4256-a49c-85366e57ac6c","Type":"ContainerDied","Data":"01ea851b44bc254aa5dc946c4a43f3dbcb3b3890adb5dccf0179c9d9d7472bac"}
Jul 27 21:24:09.388147 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.388122    2580 scope.go:117] "RemoveContainer" containerID="50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"
Jul 27 21:24:09.388603 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:24:09.388577    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77\": container with ID starting with 50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77 not found: ID does not exist" containerID="50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"
Jul 27 21:24:09.388709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.388614    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77"} err="failed to get container status \"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77\": rpc error: code = NotFound desc = could not find container \"50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77\": container with ID starting with 50762e8f3447394fd6e468ceede1bec20873676d609831229c1fe1683b7b4f77 not found: ID does not exist"
Jul 27 21:24:09.388709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.388642    2580 scope.go:117] "RemoveContainer" containerID="1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"
Jul 27 21:24:09.398185 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.398011    2580 scope.go:117] "RemoveContainer" containerID="1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"
Jul 27 21:24:09.398453 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:24:09.398379    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02\": container with ID starting with 1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02 not found: ID does not exist" containerID="1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"
Jul 27 21:24:09.398453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.398421    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02"} err="failed to get container status \"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02\": rpc error: code = NotFound desc = could not find container \"1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02\": container with ID starting with 1ed99ffb50b1af4d6e9c651bd785d404a85e4330b1a12e70dce90620779cdf02 not found: ID does not exist"
Jul 27 21:24:09.398453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.398442    2580 scope.go:117] "RemoveContainer" containerID="a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"
Jul 27 21:24:09.404189 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.404158    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:24:09.408177 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.408147    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c5876a31-7845485958-fzkvx"]
Jul 27 21:24:09.408610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.408591    2580 scope.go:117] "RemoveContainer" containerID="a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"
Jul 27 21:24:09.408925 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:24:09.408906    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf\": container with ID starting with a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf not found: ID does not exist" containerID="a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"
Jul 27 21:24:09.408994 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.408930    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf"} err="failed to get container status \"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf\": rpc error: code = NotFound desc = could not find container \"a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf\": container with ID starting with a20a1b7764a6c1ea181a4ecea7c233831428194a0aa5ebdc955e4aedaa4c37bf not found: ID does not exist"
Jul 27 21:24:09.418866 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.418831    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:24:09.423984 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.423950    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c5876a31-578bb89c69-fg424"]
Jul 27 21:24:09.433180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.433144    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:24:09.438443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:09.438411    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c5876a31-b75d6bbf6-pbcbx"]
Jul 27 21:24:10.805172 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:10.805136    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550ef780-5b08-4d03-9ca8-0c120a98be77" path="/var/lib/kubelet/pods/550ef780-5b08-4d03-9ca8-0c120a98be77/volumes"
Jul 27 21:24:10.805574 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:10.805533    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78510440-9d87-4934-ab79-e56e4b508bb0" path="/var/lib/kubelet/pods/78510440-9d87-4934-ab79-e56e4b508bb0/volumes"
Jul 27 21:24:10.805875 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:10.805862    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd51454f-202b-4256-a49c-85366e57ac6c" path="/var/lib/kubelet/pods/fd51454f-202b-4256-a49c-85366e57ac6c/volumes"
Jul 27 21:24:13.377763 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:13.377733    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:13.794525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:13.794486    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:19.461198 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:19.461160    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:19.513879 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:19.513829    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:19.727541 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:19.727444    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:19.962709 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:19.962667    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:20.084084 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:20.083985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:20.183449 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:20.183410    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:20.300116 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:20.300074    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:26.039602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:26.039511    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:26.156624 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:26.156590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:26.240306 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:26.240275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:26.401409 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:26.401371    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:32.070455 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:32.070421    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:32.171781 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:32.171744    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:38.878562 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:38.878521    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:39.187275 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:39.187147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:39.394164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:39.394109    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:39.759717 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:39.759679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:42.691139 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.691086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:42.785540 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.785495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:42.889746 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.889704    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:24:42.890178 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890160    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78510440-9d87-4934-ab79-e56e4b508bb0" containerName="payload-processing"
Jul 27 21:24:42.890290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890179    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="78510440-9d87-4934-ab79-e56e4b508bb0" containerName="payload-processing"
Jul 27 21:24:42.890290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890200    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550ef780-5b08-4d03-9ca8-0c120a98be77" containerName="payload-pre-processing"
Jul 27 21:24:42.890290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890224    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="550ef780-5b08-4d03-9ca8-0c120a98be77" containerName="payload-pre-processing"
Jul 27 21:24:42.890290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890239    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd51454f-202b-4256-a49c-85366e57ac6c" containerName="maas-api"
Jul 27 21:24:42.890290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd51454f-202b-4256-a49c-85366e57ac6c" containerName="maas-api"
Jul 27 21:24:42.890536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890362    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd51454f-202b-4256-a49c-85366e57ac6c" containerName="maas-api"
Jul 27 21:24:42.890536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890377    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="550ef780-5b08-4d03-9ca8-0c120a98be77" containerName="payload-pre-processing"
Jul 27 21:24:42.890536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.890389    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="78510440-9d87-4934-ab79-e56e4b508bb0" containerName="payload-processing"
Jul 27 21:24:42.894545 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.894518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:42.897682 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.897660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3499910e-serving-cert\""
Jul 27 21:24:42.904933 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.904888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:24:42.923041 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.923003    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:42.932639 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.932607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:42.932845 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.932690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:42.932845 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:42.932730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8fmr\" (UniqueName: \"kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.004024 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.003987    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:43.033974 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.033937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.034166 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.034025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.034166 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.034063    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8fmr\" (UniqueName: \"kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.034720 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.034698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.036805 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.036784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.043654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.043627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8fmr\" (UniqueName: \"kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr\") pod \"maas-api-e2e-mt-3499910e-66b64d77-2xwxn\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.207181 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.207136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:43.356865 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.356838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:24:43.358874 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:24:43.358842    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bb94b1_fef8_404a_ba1b_09dae9af3eed.slice/crio-ab99d348cda3d2a37f42242b2b152a75ae0eb2976705752bb11fde98453ee306 WatchSource:0}: Error finding container ab99d348cda3d2a37f42242b2b152a75ae0eb2976705752bb11fde98453ee306: Status 404 returned error can't find the container with id ab99d348cda3d2a37f42242b2b152a75ae0eb2976705752bb11fde98453ee306
Jul 27 21:24:43.360275 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.360254    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:24:43.553700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.553609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" event={"ID":"89bb94b1-fef8-404a-ba1b-09dae9af3eed","Type":"ContainerStarted","Data":"ab99d348cda3d2a37f42242b2b152a75ae0eb2976705752bb11fde98453ee306"}
Jul 27 21:24:43.602473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.602436    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:43.720459 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.720423    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:43.897450 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.897357    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:24:43.900889 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.900861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:43.903918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.903891    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3499910e\""
Jul 27 21:24:43.904049 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.903895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3499910e-dockercfg-tmzkd\""
Jul 27 21:24:43.912820 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.912795    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:24:43.944006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.943973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxw87\" (UniqueName: \"kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:43.944006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:43.944011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.045056 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.045026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxw87\" (UniqueName: \"kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.045261 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.045058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.045739 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.045710    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.057594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.057556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxw87\" (UniqueName: \"kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87\") pod \"payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.213421 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.213386    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.364990 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.364945    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:24:44.367378 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:24:44.367348    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef59f430_98f4_4bff_8e8b_77842b709d5c.slice/crio-86bdd5a40de49036ef400f7750f7c257630f5e735b80b66a6ec6f8b56fda1535 WatchSource:0}: Error finding container 86bdd5a40de49036ef400f7750f7c257630f5e735b80b66a6ec6f8b56fda1535: Status 404 returned error can't find the container with id 86bdd5a40de49036ef400f7750f7c257630f5e735b80b66a6ec6f8b56fda1535
Jul 27 21:24:44.560034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.559933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" event={"ID":"ef59f430-98f4-4bff-8e8b-77842b709d5c","Type":"ContainerStarted","Data":"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"}
Jul 27 21:24:44.560034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.560000    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" event={"ID":"ef59f430-98f4-4bff-8e8b-77842b709d5c","Type":"ContainerStarted","Data":"86bdd5a40de49036ef400f7750f7c257630f5e735b80b66a6ec6f8b56fda1535"}
Jul 27 21:24:44.560290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.560117    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:24:44.561524 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.561498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" event={"ID":"89bb94b1-fef8-404a-ba1b-09dae9af3eed","Type":"ContainerStarted","Data":"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"}
Jul 27 21:24:44.561677 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.561630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:44.593594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.589589    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" podStartSLOduration=1.589566507 podStartE2EDuration="1.589566507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:44.586056247 +0000 UTC m=+1990.359988664" watchObservedRunningTime="2026-07-27 21:24:44.589566507 +0000 UTC m=+1990.363498929"
Jul 27 21:24:44.614887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:44.614691    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" podStartSLOduration=2.227683822 podStartE2EDuration="2.614671293s" totalImagesPullingTime="386.987471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:44.614078496 +0000 UTC m=+1990.388010917" watchObservedRunningTime="2026-07-27 21:24:44.614671293 +0000 UTC m=+1990.388603714"
Jul 27 21:24:47.176945 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:47.176903    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:50.573036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:50.573005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:24:51.083452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.083409    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:24:51.087903 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.087880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.091859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.091809    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3499910e\"/\"openshift-service-ca.crt\""
Jul 27 21:24:51.091859 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.091836    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3499910e\"/\"e2e-lifecycle-model-3499910e-kserve-self-signed-certs\""
Jul 27 21:24:51.092079 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.091922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3499910e\"/\"default-dockercfg-5zk6q\""
Jul 27 21:24:51.092079 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.091846    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3499910e\"/\"kube-root-ca.crt\""
Jul 27 21:24:51.095122 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.095095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:24:51.123637 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.123603    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:51.213750 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.213965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.213965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213876    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.213965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.213965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbt2\" (UniqueName: \"kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.214120 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.213977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315002 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.314966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315191 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315191 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315083    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315191 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbt2\" (UniqueName: \"kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315191 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.315664 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.315588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.317757 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.317726    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.317920 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.317902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.323985 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.323961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbt2\" (UniqueName: \"kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2\") pod \"e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") " pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.400869 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.400770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:24:51.542881 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.542853    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:24:51.545282 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:24:51.545249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d7d3ab_9bab_4f64_8090_c4902112a206.slice/crio-54b7b850b9c1644750200cd0cea4c1477f9707893ff82c253800097fd5e43354 WatchSource:0}: Error finding container 54b7b850b9c1644750200cd0cea4c1477f9707893ff82c253800097fd5e43354: Status 404 returned error can't find the container with id 54b7b850b9c1644750200cd0cea4c1477f9707893ff82c253800097fd5e43354
Jul 27 21:24:51.599003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:51.598967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" event={"ID":"a2d7d3ab-9bab-4f64-8090-c4902112a206","Type":"ContainerStarted","Data":"54b7b850b9c1644750200cd0cea4c1477f9707893ff82c253800097fd5e43354"}
Jul 27 21:24:52.605284 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:52.605248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" event={"ID":"a2d7d3ab-9bab-4f64-8090-c4902112a206","Type":"ContainerStarted","Data":"e33362c2b06ca516595b202fedb23457efa172c60b7c3ab839680e19cd329c81"}
Jul 27 21:24:52.621889 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:52.621835    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" podStartSLOduration=1.621816607 podStartE2EDuration="1.621816607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:52.620532169 +0000 UTC m=+1998.394464590" watchObservedRunningTime="2026-07-27 21:24:52.621816607 +0000 UTC m=+1998.395749084"
Jul 27 21:24:55.569530 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:24:55.569499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:25:01.401453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:01.401420    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:25:01.401453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:01.401465    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:25:01.414240 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:01.414181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:25:01.659548 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:01.659461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:25:31.943634 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:31.943569    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:25:31.944153 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:31.944102    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" podUID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" containerName="maas-api" containerID="cri-o://3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4" gracePeriod=30
Jul 27 21:25:31.978672 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:31.978635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:32.004997 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.004957    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:25:32.005291 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.005240    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" podUID="ef59f430-98f4-4bff-8e8b-77842b709d5c" containerName="payload-pre-processing" containerID="cri-o://94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac" gracePeriod=30
Jul 27 21:25:32.079111 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.079072    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:32.179292 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.179205    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:32.258550 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.258527    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:25:32.286935 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.286909    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:25:32.384004 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.383961    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca\") pod \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") "
Jul 27 21:25:32.384004 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxw87\" (UniqueName: \"kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87\") pod \"ef59f430-98f4-4bff-8e8b-77842b709d5c\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") "
Jul 27 21:25:32.384308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8fmr\" (UniqueName: \"kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr\") pod \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") "
Jul 27 21:25:32.384308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls\") pod \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\" (UID: \"89bb94b1-fef8-404a-ba1b-09dae9af3eed\") "
Jul 27 21:25:32.384308 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume\") pod \"ef59f430-98f4-4bff-8e8b-77842b709d5c\" (UID: \"ef59f430-98f4-4bff-8e8b-77842b709d5c\") "
Jul 27 21:25:32.384522 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89bb94b1-fef8-404a-ba1b-09dae9af3eed" (UID: "89bb94b1-fef8-404a-ba1b-09dae9af3eed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:32.384663 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.384632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef59f430-98f4-4bff-8e8b-77842b709d5c" (UID: "ef59f430-98f4-4bff-8e8b-77842b709d5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:32.386368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.386346    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr" (OuterVolumeSpecName: "kube-api-access-j8fmr") pod "89bb94b1-fef8-404a-ba1b-09dae9af3eed" (UID: "89bb94b1-fef8-404a-ba1b-09dae9af3eed"). InnerVolumeSpecName "kube-api-access-j8fmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:32.386452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.386401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89bb94b1-fef8-404a-ba1b-09dae9af3eed" (UID: "89bb94b1-fef8-404a-ba1b-09dae9af3eed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:32.386452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.386419    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87" (OuterVolumeSpecName: "kube-api-access-xxw87") pod "ef59f430-98f4-4bff-8e8b-77842b709d5c" (UID: "ef59f430-98f4-4bff-8e8b-77842b709d5c"). InnerVolumeSpecName "kube-api-access-xxw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:32.485893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.485800    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89bb94b1-fef8-404a-ba1b-09dae9af3eed-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:25:32.485893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.485834    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef59f430-98f4-4bff-8e8b-77842b709d5c-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:25:32.485893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.485848    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89bb94b1-fef8-404a-ba1b-09dae9af3eed-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:25:32.485893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.485857    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxw87\" (UniqueName: \"kubernetes.io/projected/ef59f430-98f4-4bff-8e8b-77842b709d5c-kube-api-access-xxw87\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:25:32.485893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.485867    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8fmr\" (UniqueName: \"kubernetes.io/projected/89bb94b1-fef8-404a-ba1b-09dae9af3eed-kube-api-access-j8fmr\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:25:32.810975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.810883    2580 generic.go:356] "Generic (PLEG): container finished" podID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" containerID="3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4" exitCode=0
Jul 27 21:25:32.810975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.810941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"
Jul 27 21:25:32.811192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.810969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" event={"ID":"89bb94b1-fef8-404a-ba1b-09dae9af3eed","Type":"ContainerDied","Data":"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"}
Jul 27 21:25:32.811192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.811027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn" event={"ID":"89bb94b1-fef8-404a-ba1b-09dae9af3eed","Type":"ContainerDied","Data":"ab99d348cda3d2a37f42242b2b152a75ae0eb2976705752bb11fde98453ee306"}
Jul 27 21:25:32.811192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.811063    2580 scope.go:117] "RemoveContainer" containerID="3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"
Jul 27 21:25:32.812379 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.812353    2580 generic.go:356] "Generic (PLEG): container finished" podID="ef59f430-98f4-4bff-8e8b-77842b709d5c" containerID="94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac" exitCode=0
Jul 27 21:25:32.812504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.812414    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"
Jul 27 21:25:32.812504 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.812444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" event={"ID":"ef59f430-98f4-4bff-8e8b-77842b709d5c","Type":"ContainerDied","Data":"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"}
Jul 27 21:25:32.812594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.812506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn" event={"ID":"ef59f430-98f4-4bff-8e8b-77842b709d5c","Type":"ContainerDied","Data":"86bdd5a40de49036ef400f7750f7c257630f5e735b80b66a6ec6f8b56fda1535"}
Jul 27 21:25:32.823412 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.823389    2580 scope.go:117] "RemoveContainer" containerID="3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"
Jul 27 21:25:32.823720 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:25:32.823699    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4\": container with ID starting with 3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4 not found: ID does not exist" containerID="3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"
Jul 27 21:25:32.823791 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.823729    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4"} err="failed to get container status \"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4\": rpc error: code = NotFound desc = could not find container \"3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4\": container with ID starting with 3fb0c0539a5e21c19ffe0c752d698d118af58854b6561a04eacbb49514bde7a4 not found: ID does not exist"
Jul 27 21:25:32.823791 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.823748    2580 scope.go:117] "RemoveContainer" containerID="94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"
Jul 27 21:25:32.832172 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.832150    2580 scope.go:117] "RemoveContainer" containerID="94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"
Jul 27 21:25:32.832412 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.832385    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:25:32.832512 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:25:32.832500    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac\": container with ID starting with 94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac not found: ID does not exist" containerID="94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"
Jul 27 21:25:32.832571 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.832521    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac"} err="failed to get container status \"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac\": rpc error: code = NotFound desc = could not find container \"94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac\": container with ID starting with 94b7e676bc7537c246dff43ce4ea334d61365b86d8fec19b78b7e5d3be8cd2ac not found: ID does not exist"
Jul 27 21:25:32.836140 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.836115    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3499910e-66b64d77-2xwxn"]
Jul 27 21:25:32.849736 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.849705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:25:32.853206 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:32.853179    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3499910e-d889f648-phlnn"]
Jul 27 21:25:34.805358 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:34.805322    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" path="/var/lib/kubelet/pods/89bb94b1-fef8-404a-ba1b-09dae9af3eed/volumes"
Jul 27 21:25:34.805803 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:34.805675    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef59f430-98f4-4bff-8e8b-77842b709d5c" path="/var/lib/kubelet/pods/ef59f430-98f4-4bff-8e8b-77842b709d5c/volumes"
Jul 27 21:25:36.493452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:36.493417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:39.800530 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:39.800495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:44.780518 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:44.780478    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:25:44.876646 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:44.876601    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:59.800703 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:59.800607    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" podUID="a2d7d3ab-9bab-4f64-8090-c4902112a206" containerName="main" containerID="cri-o://e33362c2b06ca516595b202fedb23457efa172c60b7c3ab839680e19cd329c81" gracePeriod=45
Jul 27 21:25:59.952237 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:59.952176    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d7d3ab-9bab-4f64-8090-c4902112a206" containerID="e33362c2b06ca516595b202fedb23457efa172c60b7c3ab839680e19cd329c81" exitCode=0
Jul 27 21:25:59.952416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:25:59.952254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" event={"ID":"a2d7d3ab-9bab-4f64-8090-c4902112a206","Type":"ContainerDied","Data":"e33362c2b06ca516595b202fedb23457efa172c60b7c3ab839680e19cd329c81"}
Jul 27 21:26:00.062882 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.062807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:26:00.245958 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.245917    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.245978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxbt2\" (UniqueName: \"kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home\") pod \"a2d7d3ab-9bab-4f64-8090-c4902112a206\" (UID: \"a2d7d3ab-9bab-4f64-8090-c4902112a206\") "
Jul 27 21:26:00.246484 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246401    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:00.246539 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:00.246586 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.246530    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home" (OuterVolumeSpecName: "home") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:00.248482 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.248454    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:00.248603 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.248499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm" (OuterVolumeSpecName: "dshm") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:00.248603 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.248499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2" (OuterVolumeSpecName: "kube-api-access-sxbt2") pod "a2d7d3ab-9bab-4f64-8090-c4902112a206" (UID: "a2d7d3ab-9bab-4f64-8090-c4902112a206"). InnerVolumeSpecName "kube-api-access-sxbt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347568    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbt2\" (UniqueName: \"kubernetes.io/projected/a2d7d3ab-9bab-4f64-8090-c4902112a206-kube-api-access-sxbt2\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347595    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347605    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347612    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347619    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d7d3ab-9bab-4f64-8090-c4902112a206-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.347648 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.347626    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d7d3ab-9bab-4f64-8090-c4902112a206-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:26:00.958099 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.958067    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"
Jul 27 21:26:00.958553 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.958101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46" event={"ID":"a2d7d3ab-9bab-4f64-8090-c4902112a206","Type":"ContainerDied","Data":"54b7b850b9c1644750200cd0cea4c1477f9707893ff82c253800097fd5e43354"}
Jul 27 21:26:00.958553 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.958157    2580 scope.go:117] "RemoveContainer" containerID="e33362c2b06ca516595b202fedb23457efa172c60b7c3ab839680e19cd329c81"
Jul 27 21:26:00.974265 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.974234    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:26:00.979811 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:00.979782    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3499910e/e2e-lifecycle-model-3499910e-kserve-d8c6ff467-g7c46"]
Jul 27 21:26:02.805948 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:02.805910    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d7d3ab-9bab-4f64-8090-c4902112a206" path="/var/lib/kubelet/pods/a2d7d3ab-9bab-4f64-8090-c4902112a206/volumes"
Jul 27 21:26:14.989649 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:14.989613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:21.316821 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:21.316764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:21.815535 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:21.815497    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:27.365797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:27.365764    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:27.447508 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:27.447479    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:27.664700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:27.664608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:27.888237 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:27.888191    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:28.920310 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:28.920271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:29.023525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:29.023495    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:29.127092 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:29.127051    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:29.861083 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:29.861044    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:29.965397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:29.965361    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:30.063287 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:30.063253    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:31.520986 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:31.520941    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:35.720704 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:35.720674    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:26:35.727389 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:35.727362    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:26:37.464737 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:37.464699    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:37.568495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:37.568454    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.201013 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:38.200976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:44.348009 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:44.347964    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:44.450602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:44.450566    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:44.713283 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:44.713250    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:55.332683 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:55.332628    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:55.685080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:26:55.684999    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:16.673652 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:16.673614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:23.176650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:23.176616    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:23.379628 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:23.379588    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:23.574670 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:23.574632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:23.945192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:23.945096    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:24.503443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:24.503407    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:26.532092 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.532049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:26.628120 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.628029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:26.707443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707407    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:27:26.707823 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707809    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" containerName="maas-api"
Jul 27 21:27:26.707911 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707825    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" containerName="maas-api"
Jul 27 21:27:26.707911 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef59f430-98f4-4bff-8e8b-77842b709d5c" containerName="payload-pre-processing"
Jul 27 21:27:26.707911 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707856    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef59f430-98f4-4bff-8e8b-77842b709d5c" containerName="payload-pre-processing"
Jul 27 21:27:26.707911 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707864    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d7d3ab-9bab-4f64-8090-c4902112a206" containerName="main"
Jul 27 21:27:26.707911 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707870    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d7d3ab-9bab-4f64-8090-c4902112a206" containerName="main"
Jul 27 21:27:26.708160 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707932    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef59f430-98f4-4bff-8e8b-77842b709d5c" containerName="payload-pre-processing"
Jul 27 21:27:26.708160 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707939    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bb94b1-fef8-404a-ba1b-09dae9af3eed" containerName="maas-api"
Jul 27 21:27:26.708160 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.707946    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d7d3ab-9bab-4f64-8090-c4902112a206" containerName="main"
Jul 27 21:27:26.711417 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.711398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.714153 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.714135    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-029d89-serving-cert\""
Jul 27 21:27:26.721519 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.721497    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:27:26.736870 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.736837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:26.821883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.821852    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.821883 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.821890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.822113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.821909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm962\" (UniqueName: \"kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.908670 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.908581    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:26.922708 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.922679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.922822 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.922715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.922822 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.922732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm962\" (UniqueName: \"kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.923404 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.923386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.925260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.925239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:26.931873 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:26.931851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm962\" (UniqueName: \"kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962\") pod \"maas-api-e2e-inf-a-029d89-595df9ccf9-cf865\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:27.023043 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.023013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:27.182549 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.182521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:27:27.184523 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:27:27.184495    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ce3b08_4dd2_49fa_bc1f_f44748041b46.slice/crio-f2b42dd1d61fa7872e6d551e9d79448532cf827849deaea28430ef4b0e57af4d WatchSource:0}: Error finding container f2b42dd1d61fa7872e6d551e9d79448532cf827849deaea28430ef4b0e57af4d: Status 404 returned error can't find the container with id f2b42dd1d61fa7872e6d551e9d79448532cf827849deaea28430ef4b0e57af4d
Jul 27 21:27:27.378753 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.378720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" event={"ID":"94ce3b08-4dd2-49fa-bc1f-f44748041b46","Type":"ContainerStarted","Data":"f2b42dd1d61fa7872e6d551e9d79448532cf827849deaea28430ef4b0e57af4d"}
Jul 27 21:27:27.438064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.437986    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:27.538101 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.538061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:27.546433 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.546408    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:27:27.549961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.549935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.553040 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.553021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-029d89-dockercfg-6gb4z\""
Jul 27 21:27:27.553492 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.553475    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-029d89\""
Jul 27 21:27:27.564312 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.564284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:27:27.627750 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.627719    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57hj\" (UniqueName: \"kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.627855 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.627794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.729255 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.729194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n57hj\" (UniqueName: \"kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.729448 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.729311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.729982 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.729956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.740730 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.740700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57hj\" (UniqueName: \"kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj\") pod \"payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.863701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.863670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:27.999962 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:27.999937    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:27:28.001946 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:27:28.001926    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b0da41_502f_40cc_a363_79f0026d88fa.slice/crio-724413764dae80015723557e3d135d0f2f8a4bae23c4025f59b7f5f771d13661 WatchSource:0}: Error finding container 724413764dae80015723557e3d135d0f2f8a4bae23c4025f59b7f5f771d13661: Status 404 returned error can't find the container with id 724413764dae80015723557e3d135d0f2f8a4bae23c4025f59b7f5f771d13661
Jul 27 21:27:28.385702 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.385614    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" event={"ID":"c5b0da41-502f-40cc-a363-79f0026d88fa","Type":"ContainerStarted","Data":"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"}
Jul 27 21:27:28.385702 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.385675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" event={"ID":"c5b0da41-502f-40cc-a363-79f0026d88fa","Type":"ContainerStarted","Data":"724413764dae80015723557e3d135d0f2f8a4bae23c4025f59b7f5f771d13661"}
Jul 27 21:27:28.385702 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.385704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:28.387167 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.387139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" event={"ID":"94ce3b08-4dd2-49fa-bc1f-f44748041b46","Type":"ContainerStarted","Data":"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"}
Jul 27 21:27:28.387325 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.387233    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:28.439655 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:28.439609    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" podStartSLOduration=1.43959593 podStartE2EDuration="1.43959593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:28.437032374 +0000 UTC m=+2154.210964794" watchObservedRunningTime="2026-07-27 21:27:28.43959593 +0000 UTC m=+2154.213528387"
Jul 27 21:27:31.012161 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:31.012121    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:31.384140 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:31.384030    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:31.813631 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:31.813590    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:31.915965 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:31.915922    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:34.398436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.398408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:27:34.429707 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.429655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" podStartSLOduration=8.05196475 podStartE2EDuration="8.429638084s" totalImagesPullingTime="377.673334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:28.482992183 +0000 UTC m=+2154.256924603" watchObservedRunningTime="2026-07-27 21:27:34.429638084 +0000 UTC m=+2160.203570504"
Jul 27 21:27:34.665326 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.665235    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:34.805838 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.805810    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:27:34.809516 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.809493    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.816046 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.815721    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ccc682-serving-cert\""
Jul 27 21:27:34.839707 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.839679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:27:34.845643 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.845613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:34.895071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.895037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfjw\" (UniqueName: \"kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.895280 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.895184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.895347 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.895331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.980186 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.980155    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:34.996356 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.996328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfjw\" (UniqueName: \"kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.996463 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.996393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.996463 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.996433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:34.996545 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:27:34.996532    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-serving-cert: secret "maas-api-e2e-inf-b-ccc682-serving-cert" not found
Jul 27 21:27:34.996608 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:27:34.996600    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls podName:f77431d6-3b35-4cac-9e3c-f6449b4dd915 nodeName:}" failed. No retries permitted until 2026-07-27 21:27:35.496585294 +0000 UTC m=+2161.270517692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls") pod "maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" (UID: "f77431d6-3b35-4cac-9e3c-f6449b4dd915") : secret "maas-api-e2e-inf-b-ccc682-serving-cert" not found
Jul 27 21:27:34.997034 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:34.997015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:35.016769 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.016746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfjw\" (UniqueName: \"kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:35.185925 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.185884    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:35.501234 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.501188    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:35.503743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.503713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") pod \"maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:35.719454 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.719412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:35.906071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.905901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:27:35.908805 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:27:35.908770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77431d6_3b35_4cac_9e3c_f6449b4dd915.slice/crio-490e252f1f90690835149cc493b83b8d12eea224d5b1b2ae6ce35bc17d6466ae WatchSource:0}: Error finding container 490e252f1f90690835149cc493b83b8d12eea224d5b1b2ae6ce35bc17d6466ae: Status 404 returned error can't find the container with id 490e252f1f90690835149cc493b83b8d12eea224d5b1b2ae6ce35bc17d6466ae
Jul 27 21:27:35.938720 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:35.938683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:36.086630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:36.086537    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:36.427371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:36.427280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" event={"ID":"f77431d6-3b35-4cac-9e3c-f6449b4dd915","Type":"ContainerStarted","Data":"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"}
Jul 27 21:27:36.427371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:36.427333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" event={"ID":"f77431d6-3b35-4cac-9e3c-f6449b4dd915","Type":"ContainerStarted","Data":"490e252f1f90690835149cc493b83b8d12eea224d5b1b2ae6ce35bc17d6466ae"}
Jul 27 21:27:36.427371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:36.427364    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:36.461557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:36.461507    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" podStartSLOduration=2.10789141 podStartE2EDuration="2.461490808s" totalImagesPullingTime="353.599398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:36.461037692 +0000 UTC m=+2162.234970112" watchObservedRunningTime="2026-07-27 21:27:36.461490808 +0000 UTC m=+2162.235423228"
Jul 27 21:27:39.048630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:39.048589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:39.395046 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:39.394959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:27:42.437561 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:42.437531    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:27:48.911931 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.911894    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:27:48.915615 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.915598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:48.918578 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.918554    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-029d89\"/\"default-dockercfg-kb4zn\""
Jul 27 21:27:48.919708 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.919683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-029d89\"/\"openshift-service-ca.crt\""
Jul 27 21:27:48.919819 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.919717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-029d89\"/\"test-model-029d89-kserve-self-signed-certs\""
Jul 27 21:27:48.919819 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.919769    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-029d89\"/\"kube-root-ca.crt\""
Jul 27 21:27:48.925275 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.925254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:27:48.936440 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:48.936408    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:49.018581 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.018581 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.018802 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.018802 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.018802 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.018802 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.018739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwph\" (UniqueName: \"kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120291 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwph\" (UniqueName: \"kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120713 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120713 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120869 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.120917 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.120892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.122857 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.122825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.123085 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.123066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.129806 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.129779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwph\" (UniqueName: \"kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph\") pod \"test-model-029d89-kserve-5b848fcb65-dt49g\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") " pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.226472 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.226432    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:49.373649 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.373620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:27:49.375688 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:27:49.375662    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60019236_225e_44d8_938e_3e601af8ac96.slice/crio-29d639e4b66192152c9cf39dfb904f0728959b5a3cd1eb397e8c061e90b87b4a WatchSource:0}: Error finding container 29d639e4b66192152c9cf39dfb904f0728959b5a3cd1eb397e8c061e90b87b4a: Status 404 returned error can't find the container with id 29d639e4b66192152c9cf39dfb904f0728959b5a3cd1eb397e8c061e90b87b4a
Jul 27 21:27:49.496756 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.496644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" event={"ID":"60019236-225e-44d8-938e-3e601af8ac96","Type":"ContainerStarted","Data":"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"}
Jul 27 21:27:49.496756 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.496690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" event={"ID":"60019236-225e-44d8-938e-3e601af8ac96","Type":"ContainerStarted","Data":"29d639e4b66192152c9cf39dfb904f0728959b5a3cd1eb397e8c061e90b87b4a"}
Jul 27 21:27:49.514567 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:49.514486    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" podStartSLOduration=1.514467988 podStartE2EDuration="1.514467988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:49.514299219 +0000 UTC m=+2175.288231640" watchObservedRunningTime="2026-07-27 21:27:49.514467988 +0000 UTC m=+2175.288400410"
Jul 27 21:27:53.967499 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:53.967462    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:54.068247 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:54.068194    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:59.226959 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:59.226919    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:59.227417 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:59.227053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:59.239870 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:59.239841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:27:59.558045 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:27:59.557957    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:28:10.910787 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:28:10.910743    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:53.819660 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:28:53.819627    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:28:53.856469 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:28:53.856430    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:55.352327 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:28:55.352241    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:05.799743 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:05.799708    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:29:05.800298 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:05.799870    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" podUID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" containerName="maas-api" containerID="cri-o://c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b" gracePeriod=30
Jul 27 21:29:05.867790 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:05.867750    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:05.962843 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:05.962791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:29:05.963077 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:05.963037    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" podUID="c5b0da41-502f-40cc-a363-79f0026d88fa" containerName="payload-pre-processing" containerID="cri-o://13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f" gracePeriod=30
Jul 27 21:29:06.031488 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.031457    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:06.091117 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.091086    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:29:06.178332 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.178297    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:06.212707 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.212684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:29:06.255076 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca\") pod \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") "
Jul 27 21:29:06.255466 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255444    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume\") pod \"c5b0da41-502f-40cc-a363-79f0026d88fa\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") "
Jul 27 21:29:06.255661 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm962\" (UniqueName: \"kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962\") pod \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") "
Jul 27 21:29:06.255755 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255737    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n57hj\" (UniqueName: \"kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj\") pod \"c5b0da41-502f-40cc-a363-79f0026d88fa\" (UID: \"c5b0da41-502f-40cc-a363-79f0026d88fa\") "
Jul 27 21:29:06.255815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255773    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls\") pod \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\" (UID: \"94ce3b08-4dd2-49fa-bc1f-f44748041b46\") "
Jul 27 21:29:06.255815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255679    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94ce3b08-4dd2-49fa-bc1f-f44748041b46" (UID: "94ce3b08-4dd2-49fa-bc1f-f44748041b46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:06.255923 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.255861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5b0da41-502f-40cc-a363-79f0026d88fa" (UID: "c5b0da41-502f-40cc-a363-79f0026d88fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:06.256122 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.256089    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ce3b08-4dd2-49fa-bc1f-f44748041b46-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:06.256122 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.256114    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b0da41-502f-40cc-a363-79f0026d88fa-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:06.258194 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.258167    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962" (OuterVolumeSpecName: "kube-api-access-wm962") pod "94ce3b08-4dd2-49fa-bc1f-f44748041b46" (UID: "94ce3b08-4dd2-49fa-bc1f-f44748041b46"). InnerVolumeSpecName "kube-api-access-wm962". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:06.258480 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.258459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94ce3b08-4dd2-49fa-bc1f-f44748041b46" (UID: "94ce3b08-4dd2-49fa-bc1f-f44748041b46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:06.258570 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.258488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj" (OuterVolumeSpecName: "kube-api-access-n57hj") pod "c5b0da41-502f-40cc-a363-79f0026d88fa" (UID: "c5b0da41-502f-40cc-a363-79f0026d88fa"). InnerVolumeSpecName "kube-api-access-n57hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:06.356563 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.356477    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm962\" (UniqueName: \"kubernetes.io/projected/94ce3b08-4dd2-49fa-bc1f-f44748041b46-kube-api-access-wm962\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:06.356563 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.356505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57hj\" (UniqueName: \"kubernetes.io/projected/c5b0da41-502f-40cc-a363-79f0026d88fa-kube-api-access-n57hj\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:06.356563 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.356514    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94ce3b08-4dd2-49fa-bc1f-f44748041b46-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:06.885889 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.885853    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5b0da41-502f-40cc-a363-79f0026d88fa" containerID="13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f" exitCode=0
Jul 27 21:29:06.886361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.885918    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"
Jul 27 21:29:06.886361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.885944    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" event={"ID":"c5b0da41-502f-40cc-a363-79f0026d88fa","Type":"ContainerDied","Data":"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"}
Jul 27 21:29:06.886361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.886016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq" event={"ID":"c5b0da41-502f-40cc-a363-79f0026d88fa","Type":"ContainerDied","Data":"724413764dae80015723557e3d135d0f2f8a4bae23c4025f59b7f5f771d13661"}
Jul 27 21:29:06.886361 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.886051    2580 scope.go:117] "RemoveContainer" containerID="13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"
Jul 27 21:29:06.887525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.887268    2580 generic.go:356] "Generic (PLEG): container finished" podID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" containerID="c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b" exitCode=0
Jul 27 21:29:06.887525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.887326    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"
Jul 27 21:29:06.887525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.887317    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" event={"ID":"94ce3b08-4dd2-49fa-bc1f-f44748041b46","Type":"ContainerDied","Data":"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"}
Jul 27 21:29:06.887525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.887451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865" event={"ID":"94ce3b08-4dd2-49fa-bc1f-f44748041b46","Type":"ContainerDied","Data":"f2b42dd1d61fa7872e6d551e9d79448532cf827849deaea28430ef4b0e57af4d"}
Jul 27 21:29:06.895290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.895243    2580 scope.go:117] "RemoveContainer" containerID="13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"
Jul 27 21:29:06.895563 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:06.895546    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f\": container with ID starting with 13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f not found: ID does not exist" containerID="13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"
Jul 27 21:29:06.895621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.895572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f"} err="failed to get container status \"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f\": rpc error: code = NotFound desc = could not find container \"13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f\": container with ID starting with 13a40cad26482e46749e9d5e20421ee5cdaa611a224eb8454648495d1225689f not found: ID does not exist"
Jul 27 21:29:06.895621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.895588    2580 scope.go:117] "RemoveContainer" containerID="c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"
Jul 27 21:29:06.903537 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.903514    2580 scope.go:117] "RemoveContainer" containerID="c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"
Jul 27 21:29:06.903805 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:06.903785    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b\": container with ID starting with c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b not found: ID does not exist" containerID="c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"
Jul 27 21:29:06.903863 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.903812    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b"} err="failed to get container status \"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b\": rpc error: code = NotFound desc = could not find container \"c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b\": container with ID starting with c8c6f2303b6b92900ebe299b3e59a193724a56571bdfa32a7b3994c1b30a2f6b not found: ID does not exist"
Jul 27 21:29:06.913666 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.913637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:29:06.925568 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.925541    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-029d89-595df9ccf9-cf865"]
Jul 27 21:29:06.947628 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.947596    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:29:06.957291 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:06.957264    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-029d89-5d676999d4-vm2pq"]
Jul 27 21:29:08.806191 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:08.806150    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" path="/var/lib/kubelet/pods/94ce3b08-4dd2-49fa-bc1f-f44748041b46/volumes"
Jul 27 21:29:08.806607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:08.806588    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b0da41-502f-40cc-a363-79f0026d88fa" path="/var/lib/kubelet/pods/c5b0da41-502f-40cc-a363-79f0026d88fa/volumes"
Jul 27 21:29:08.839388 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:08.839348    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" podUID="60019236-225e-44d8-938e-3e601af8ac96" containerName="main" containerID="cri-o://8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854" gracePeriod=45
Jul 27 21:29:09.095821 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.095788    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:29:09.179006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.178969    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179006    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179047    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdwph\" (UniqueName: \"kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179142    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm\") pod \"60019236-225e-44d8-938e-3e601af8ac96\" (UID: \"60019236-225e-44d8-938e-3e601af8ac96\") "
Jul 27 21:29:09.179493 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache" (OuterVolumeSpecName: "model-cache") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:09.179493 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home" (OuterVolumeSpecName: "home") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:09.179493 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:09.179613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179523    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.179613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179538    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.179613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.179548    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.181626 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.181602    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm" (OuterVolumeSpecName: "dshm") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:09.181722 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.181647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph" (OuterVolumeSpecName: "kube-api-access-bdwph") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "kube-api-access-bdwph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:09.181722 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.181701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60019236-225e-44d8-938e-3e601af8ac96" (UID: "60019236-225e-44d8-938e-3e601af8ac96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:09.280731 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.280699    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60019236-225e-44d8-938e-3e601af8ac96-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.280731 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.280730    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60019236-225e-44d8-938e-3e601af8ac96-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.280731 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.280740    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwph\" (UniqueName: \"kubernetes.io/projected/60019236-225e-44d8-938e-3e601af8ac96-kube-api-access-bdwph\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:09.906185 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.906150    2580 generic.go:356] "Generic (PLEG): container finished" podID="60019236-225e-44d8-938e-3e601af8ac96" containerID="8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854" exitCode=0
Jul 27 21:29:09.906689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.906191    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"
Jul 27 21:29:09.906689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.906248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" event={"ID":"60019236-225e-44d8-938e-3e601af8ac96","Type":"ContainerDied","Data":"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"}
Jul 27 21:29:09.906689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.906293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g" event={"ID":"60019236-225e-44d8-938e-3e601af8ac96","Type":"ContainerDied","Data":"29d639e4b66192152c9cf39dfb904f0728959b5a3cd1eb397e8c061e90b87b4a"}
Jul 27 21:29:09.906689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.906313    2580 scope.go:117] "RemoveContainer" containerID="8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"
Jul 27 21:29:09.916354 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.916337    2580 scope.go:117] "RemoveContainer" containerID="8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"
Jul 27 21:29:09.916644 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:09.916626    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854\": container with ID starting with 8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854 not found: ID does not exist" containerID="8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"
Jul 27 21:29:09.916693 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.916656    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854"} err="failed to get container status \"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854\": rpc error: code = NotFound desc = could not find container \"8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854\": container with ID starting with 8ab91ed085c2eca9d02f9d593a7f841436335fb7fefe5657b37525808f36e854 not found: ID does not exist"
Jul 27 21:29:09.932017 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.931987    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:29:09.935891 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:09.935867    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-029d89/test-model-029d89-kserve-5b848fcb65-dt49g"]
Jul 27 21:29:10.114055 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:10.114017    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:10.571347 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:10.571311    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:10.805478 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:10.805432    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60019236-225e-44d8-938e-3e601af8ac96" path="/var/lib/kubelet/pods/60019236-225e-44d8-938e-3e601af8ac96/volumes"
Jul 27 21:29:17.810972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:17.810915    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:17.977206 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:17.977166    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:18.314577 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:18.314535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:18.474750 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:18.474711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:23.164677 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:23.164645    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:29.186955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.186681    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:29:29.186955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.186946    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" podUID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" containerName="maas-api" containerID="cri-o://13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f" gracePeriod=30
Jul 27 21:29:29.236784 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.236748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:29.445862 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.445791    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:29:29.533711 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.533676    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:29.568977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.568938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqfjw\" (UniqueName: \"kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw\") pod \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") "
Jul 27 21:29:29.569166 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.569070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca\") pod \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") "
Jul 27 21:29:29.569166 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.569090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") pod \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\" (UID: \"f77431d6-3b35-4cac-9e3c-f6449b4dd915\") "
Jul 27 21:29:29.569511 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.569485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f77431d6-3b35-4cac-9e3c-f6449b4dd915" (UID: "f77431d6-3b35-4cac-9e3c-f6449b4dd915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:29.571424 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.571396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw" (OuterVolumeSpecName: "kube-api-access-fqfjw") pod "f77431d6-3b35-4cac-9e3c-f6449b4dd915" (UID: "f77431d6-3b35-4cac-9e3c-f6449b4dd915"). InnerVolumeSpecName "kube-api-access-fqfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:29.571513 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.571439    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f77431d6-3b35-4cac-9e3c-f6449b4dd915" (UID: "f77431d6-3b35-4cac-9e3c-f6449b4dd915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:29.670426 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.670392    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77431d6-3b35-4cac-9e3c-f6449b4dd915-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:29.670426 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.670422    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f77431d6-3b35-4cac-9e3c-f6449b4dd915-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:29.670639 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:29.670432    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfjw\" (UniqueName: \"kubernetes.io/projected/f77431d6-3b35-4cac-9e3c-f6449b4dd915-kube-api-access-fqfjw\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:29:30.010224 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.010188    2580 generic.go:356] "Generic (PLEG): container finished" podID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" containerID="13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f" exitCode=0
Jul 27 21:29:30.010406 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.010260    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"
Jul 27 21:29:30.010406 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.010274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" event={"ID":"f77431d6-3b35-4cac-9e3c-f6449b4dd915","Type":"ContainerDied","Data":"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"}
Jul 27 21:29:30.010406 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.010340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6" event={"ID":"f77431d6-3b35-4cac-9e3c-f6449b4dd915","Type":"ContainerDied","Data":"490e252f1f90690835149cc493b83b8d12eea224d5b1b2ae6ce35bc17d6466ae"}
Jul 27 21:29:30.010406 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.010370    2580 scope.go:117] "RemoveContainer" containerID="13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"
Jul 27 21:29:30.020042 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.020021    2580 scope.go:117] "RemoveContainer" containerID="13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"
Jul 27 21:29:30.020370 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:30.020351    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f\": container with ID starting with 13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f not found: ID does not exist" containerID="13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"
Jul 27 21:29:30.020434 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.020378    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f"} err="failed to get container status \"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f\": rpc error: code = NotFound desc = could not find container \"13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f\": container with ID starting with 13f1b39ab137b2d8e3b16cfd5b5dfc9f8543e07235089a7b70c38835405d875f not found: ID does not exist"
Jul 27 21:29:30.036824 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.036789    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:29:30.044998 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.044964    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ccc682-6dddb88d45-lnjp6"]
Jul 27 21:29:30.811017 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:30.810980    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" path="/var/lib/kubelet/pods/f77431d6-3b35-4cac-9e3c-f6449b4dd915/volumes"
Jul 27 21:29:32.995490 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:32.995448    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:33.684100 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:33.684061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:34.180393 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:34.180357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:40.411493 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:40.411456    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:40.513356 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:40.513310    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:40.705343 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:40.705246    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:40.955711 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:40.955683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:41.288255 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:41.288144    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:41.359173 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:41.359112    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:41.709975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:41.709945    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:41.916853 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:41.916810    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:45.141750 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.141716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:45.232543 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232506    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:29:45.232889 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232877    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" containerName="maas-api"
Jul 27 21:29:45.232936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232891    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" containerName="maas-api"
Jul 27 21:29:45.232936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232904    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" containerName="maas-api"
Jul 27 21:29:45.232936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232910    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" containerName="maas-api"
Jul 27 21:29:45.232936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232918    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60019236-225e-44d8-938e-3e601af8ac96" containerName="main"
Jul 27 21:29:45.232936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232923    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60019236-225e-44d8-938e-3e601af8ac96" containerName="main"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232941    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b0da41-502f-40cc-a363-79f0026d88fa" containerName="payload-pre-processing"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.232947    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b0da41-502f-40cc-a363-79f0026d88fa" containerName="payload-pre-processing"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.233007    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60019236-225e-44d8-938e-3e601af8ac96" containerName="main"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.233014    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77431d6-3b35-4cac-9e3c-f6449b4dd915" containerName="maas-api"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.233023    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ce3b08-4dd2-49fa-bc1f-f44748041b46" containerName="maas-api"
Jul 27 21:29:45.233089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.233030    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b0da41-502f-40cc-a363-79f0026d88fa" containerName="payload-pre-processing"
Jul 27 21:29:45.236489 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.236468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.239067 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.239042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9d44e2-serving-cert\""
Jul 27 21:29:45.240202 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.240174    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:45.246900 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.246875    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:29:45.341673 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.341635    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:45.417898 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.417794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.417898 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.417841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmk5f\" (UniqueName: \"kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.418113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.417949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.519168 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.519134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.519168 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.519184    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmk5f\" (UniqueName: \"kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.519461 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.519271    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.519461 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:45.519391    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-serving-cert: secret "maas-api-e2e-api-a-9d44e2-serving-cert" not found
Jul 27 21:29:45.519570 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:29:45.519472    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls podName:88e85b0b-dfef-4354-9deb-1e4795e9b33a nodeName:}" failed. No retries permitted until 2026-07-27 21:29:46.019449664 +0000 UTC m=+2291.793382064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls") pod "maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" (UID: "88e85b0b-dfef-4354-9deb-1e4795e9b33a") : secret "maas-api-e2e-api-a-9d44e2-serving-cert" not found
Jul 27 21:29:45.519782 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.519759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:45.530169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:45.530135    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmk5f\" (UniqueName: \"kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:46.020041 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.019990    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:46.024018 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.023991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:46.026613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.026595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") pod \"maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:46.148391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.148354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:46.263355 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.263317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:46.338026 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.337994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:29:46.340946 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:29:46.340913    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e85b0b_dfef_4354_9deb_1e4795e9b33a.slice/crio-6c759e1218c3452ddb87972b864e81692718bb483b75d438a4bcb5a424f50fe6 WatchSource:0}: Error finding container 6c759e1218c3452ddb87972b864e81692718bb483b75d438a4bcb5a424f50fe6: Status 404 returned error can't find the container with id 6c759e1218c3452ddb87972b864e81692718bb483b75d438a4bcb5a424f50fe6
Jul 27 21:29:46.342621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.342603    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:29:46.381416 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.381373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:46.477595 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.477561    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:29:46.481778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.481761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.486897 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.486876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9d44e2-dockercfg-gthb6\""
Jul 27 21:29:46.487036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.486974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9d44e2\""
Jul 27 21:29:46.500469 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.500439    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:29:46.526047 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.526007    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:29:46.529542 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.529518    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.545528 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.545501    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:29:46.630527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.630433    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.630527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.630507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.630723 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.630537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffms\" (UniqueName: \"kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.630723 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.630567    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rxq\" (UniqueName: \"kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.731645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.731606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.731823 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.731657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.731823 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.731676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffms\" (UniqueName: \"kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.731823 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.731697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rxq\" (UniqueName: \"kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.732634 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.732608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.732684 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.732608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.745816 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.745792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rxq\" (UniqueName: \"kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq\") pod \"payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.755975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.755952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffms\" (UniqueName: \"kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms\") pod \"payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") " pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.792246 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.792195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:46.839385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.839351    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:46.968013 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:46.967954    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:29:46.971047 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:29:46.971004    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d59c9f_c06e_4789_b523_e1a01cf83bba.slice/crio-65a8dedf506df0c374fdbd692a6a2011fc690cd9a05deb4ecc61a79caa771273 WatchSource:0}: Error finding container 65a8dedf506df0c374fdbd692a6a2011fc690cd9a05deb4ecc61a79caa771273: Status 404 returned error can't find the container with id 65a8dedf506df0c374fdbd692a6a2011fc690cd9a05deb4ecc61a79caa771273
Jul 27 21:29:47.014810 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.014776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:29:47.029493 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:29:47.029463    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb0f98c_e26c_4154_b8d4_ddea93c13562.slice/crio-205469be5129288688b3850304f6a3e65eabe149d7c48d18294af599ae23a2c4 WatchSource:0}: Error finding container 205469be5129288688b3850304f6a3e65eabe149d7c48d18294af599ae23a2c4: Status 404 returned error can't find the container with id 205469be5129288688b3850304f6a3e65eabe149d7c48d18294af599ae23a2c4
Jul 27 21:29:47.095936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.095901    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" event={"ID":"e9d59c9f-c06e-4789-b523-e1a01cf83bba","Type":"ContainerStarted","Data":"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"}
Jul 27 21:29:47.096124 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.095963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" event={"ID":"e9d59c9f-c06e-4789-b523-e1a01cf83bba","Type":"ContainerStarted","Data":"65a8dedf506df0c374fdbd692a6a2011fc690cd9a05deb4ecc61a79caa771273"}
Jul 27 21:29:47.096124 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.096082    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:47.097082 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.097051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" event={"ID":"adb0f98c-e26c-4154-b8d4-ddea93c13562","Type":"ContainerStarted","Data":"205469be5129288688b3850304f6a3e65eabe149d7c48d18294af599ae23a2c4"}
Jul 27 21:29:47.098593 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.098566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" event={"ID":"88e85b0b-dfef-4354-9deb-1e4795e9b33a","Type":"ContainerStarted","Data":"c5369d27d78341b60989b6d5bc677b59af9e5b0e210806b6b6b392838f1c1bad"}
Jul 27 21:29:47.098716 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.098607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" event={"ID":"88e85b0b-dfef-4354-9deb-1e4795e9b33a","Type":"ContainerStarted","Data":"6c759e1218c3452ddb87972b864e81692718bb483b75d438a4bcb5a424f50fe6"}
Jul 27 21:29:47.098716 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.098689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:47.127967 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.127913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" podStartSLOduration=1.127898295 podStartE2EDuration="1.127898295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:47.126694574 +0000 UTC m=+2292.900626997" watchObservedRunningTime="2026-07-27 21:29:47.127898295 +0000 UTC m=+2292.901830714"
Jul 27 21:29:47.150182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:47.150061    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" podStartSLOduration=1.7493818110000001 podStartE2EDuration="2.150043129s" totalImagesPullingTime="400.661318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:47.149567697 +0000 UTC m=+2292.923500120" watchObservedRunningTime="2026-07-27 21:29:47.150043129 +0000 UTC m=+2292.923975549"
Jul 27 21:29:48.105301 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:48.105265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" event={"ID":"adb0f98c-e26c-4154-b8d4-ddea93c13562","Type":"ContainerStarted","Data":"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"}
Jul 27 21:29:48.105469 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:48.105348    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:29:48.128610 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:48.128559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" podStartSLOduration=2.128544453 podStartE2EDuration="2.128544453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:48.126405494 +0000 UTC m=+2293.900337918" watchObservedRunningTime="2026-07-27 21:29:48.128544453 +0000 UTC m=+2293.902476919"
Jul 27 21:29:49.640682 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:49.640642    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:50.208073 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:50.208041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:50.569340 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:50.569230    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:50.670289 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:50.670257    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:53.111411 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:53.111382    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:29:53.350715 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:53.350679    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:53.528390 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:53.528333    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:53.726761 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:53.726729    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:53.868336 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:53.868207    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:54.483206 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:54.483167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:54.720038 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:54.719962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:57.729186 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:57.729136    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:58.107699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:58.107620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:29:59.112617 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:29:59.112584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:30:22.159116 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.159075    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:30:22.159706 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.159375    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" podUID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" containerName="maas-api" containerID="cri-o://c5369d27d78341b60989b6d5bc677b59af9e5b0e210806b6b6b392838f1c1bad" gracePeriod=30
Jul 27 21:30:22.185169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.185137    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:22.220498 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.220464    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:30:22.220733 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.220708    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" podUID="adb0f98c-e26c-4154-b8d4-ddea93c13562" containerName="payload-processing" containerID="cri-o://044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645" gracePeriod=30
Jul 27 21:30:22.227924 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.227891    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:30:22.228186 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.228140    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" podUID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" containerName="payload-pre-processing" containerID="cri-o://5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7" gracePeriod=30
Jul 27 21:30:22.250423 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.250303    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:22.278284 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.278245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" event={"ID":"88e85b0b-dfef-4354-9deb-1e4795e9b33a","Type":"ContainerDied","Data":"c5369d27d78341b60989b6d5bc677b59af9e5b0e210806b6b6b392838f1c1bad"}
Jul 27 21:30:22.278444 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.278202    2580 generic.go:356] "Generic (PLEG): container finished" podID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" containerID="c5369d27d78341b60989b6d5bc677b59af9e5b0e210806b6b6b392838f1c1bad" exitCode=0
Jul 27 21:30:22.353267 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.353231    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:22.508158 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.508132    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:30:22.521258 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.521208    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:30:22.544268 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.544240    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:30:22.662792 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662750    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmk5f\" (UniqueName: \"kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f\") pod \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662806    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffms\" (UniqueName: \"kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms\") pod \"adb0f98c-e26c-4154-b8d4-ddea93c13562\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662850    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") pod \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume\") pod \"adb0f98c-e26c-4154-b8d4-ddea93c13562\" (UID: \"adb0f98c-e26c-4154-b8d4-ddea93c13562\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca\") pod \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\" (UID: \"88e85b0b-dfef-4354-9deb-1e4795e9b33a\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5rxq\" (UniqueName: \"kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq\") pod \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") "
Jul 27 21:30:22.663071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.662949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume\") pod \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\" (UID: \"e9d59c9f-c06e-4789-b523-e1a01cf83bba\") "
Jul 27 21:30:22.663460 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663345    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88e85b0b-dfef-4354-9deb-1e4795e9b33a" (UID: "88e85b0b-dfef-4354-9deb-1e4795e9b33a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:22.663460 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adb0f98c-e26c-4154-b8d4-ddea93c13562" (UID: "adb0f98c-e26c-4154-b8d4-ddea93c13562"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:22.663569 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663484    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9d59c9f-c06e-4789-b523-e1a01cf83bba" (UID: "e9d59c9f-c06e-4789-b523-e1a01cf83bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:22.663790 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663692    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb0f98c-e26c-4154-b8d4-ddea93c13562-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.663790 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663713    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e85b0b-dfef-4354-9deb-1e4795e9b33a-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.663790 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.663727    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9d59c9f-c06e-4789-b523-e1a01cf83bba-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.665399 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.665360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq" (OuterVolumeSpecName: "kube-api-access-b5rxq") pod "e9d59c9f-c06e-4789-b523-e1a01cf83bba" (UID: "e9d59c9f-c06e-4789-b523-e1a01cf83bba"). InnerVolumeSpecName "kube-api-access-b5rxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:22.665540 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.665411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms" (OuterVolumeSpecName: "kube-api-access-6ffms") pod "adb0f98c-e26c-4154-b8d4-ddea93c13562" (UID: "adb0f98c-e26c-4154-b8d4-ddea93c13562"). InnerVolumeSpecName "kube-api-access-6ffms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:22.665618 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.665594    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f" (OuterVolumeSpecName: "kube-api-access-fmk5f") pod "88e85b0b-dfef-4354-9deb-1e4795e9b33a" (UID: "88e85b0b-dfef-4354-9deb-1e4795e9b33a"). InnerVolumeSpecName "kube-api-access-fmk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:22.665719 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.665695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88e85b0b-dfef-4354-9deb-1e4795e9b33a" (UID: "88e85b0b-dfef-4354-9deb-1e4795e9b33a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:30:22.764370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.764329    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmk5f\" (UniqueName: \"kubernetes.io/projected/88e85b0b-dfef-4354-9deb-1e4795e9b33a-kube-api-access-fmk5f\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.764370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.764367    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffms\" (UniqueName: \"kubernetes.io/projected/adb0f98c-e26c-4154-b8d4-ddea93c13562-kube-api-access-6ffms\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.764370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.764378    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e85b0b-dfef-4354-9deb-1e4795e9b33a-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:22.764593 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:22.764387    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5rxq\" (UniqueName: \"kubernetes.io/projected/e9d59c9f-c06e-4789-b523-e1a01cf83bba-kube-api-access-b5rxq\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:30:23.285380 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.285346    2580 generic.go:356] "Generic (PLEG): container finished" podID="adb0f98c-e26c-4154-b8d4-ddea93c13562" containerID="044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645" exitCode=0
Jul 27 21:30:23.285797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.285409    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"
Jul 27 21:30:23.285797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.285433    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" event={"ID":"adb0f98c-e26c-4154-b8d4-ddea93c13562","Type":"ContainerDied","Data":"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"}
Jul 27 21:30:23.285797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.285507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv" event={"ID":"adb0f98c-e26c-4154-b8d4-ddea93c13562","Type":"ContainerDied","Data":"205469be5129288688b3850304f6a3e65eabe149d7c48d18294af599ae23a2c4"}
Jul 27 21:30:23.285797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.285537    2580 scope.go:117] "RemoveContainer" containerID="044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"
Jul 27 21:30:23.286933 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.286914    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"
Jul 27 21:30:23.287052 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.286957    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4" event={"ID":"88e85b0b-dfef-4354-9deb-1e4795e9b33a","Type":"ContainerDied","Data":"6c759e1218c3452ddb87972b864e81692718bb483b75d438a4bcb5a424f50fe6"}
Jul 27 21:30:23.288770 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.288744    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" containerID="5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7" exitCode=0
Jul 27 21:30:23.288887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.288817    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"
Jul 27 21:30:23.288887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.288821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" event={"ID":"e9d59c9f-c06e-4789-b523-e1a01cf83bba","Type":"ContainerDied","Data":"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"}
Jul 27 21:30:23.288887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.288875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz" event={"ID":"e9d59c9f-c06e-4789-b523-e1a01cf83bba","Type":"ContainerDied","Data":"65a8dedf506df0c374fdbd692a6a2011fc690cd9a05deb4ecc61a79caa771273"}
Jul 27 21:30:23.296019 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.295999    2580 scope.go:117] "RemoveContainer" containerID="044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"
Jul 27 21:30:23.296379 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:30:23.296359    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645\": container with ID starting with 044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645 not found: ID does not exist" containerID="044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"
Jul 27 21:30:23.296453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.296389    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645"} err="failed to get container status \"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645\": rpc error: code = NotFound desc = could not find container \"044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645\": container with ID starting with 044df3f444a6fa73cfd5645b6daa370071d090eb25564314e49e4e1ccdb54645 not found: ID does not exist"
Jul 27 21:30:23.296453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.296407    2580 scope.go:117] "RemoveContainer" containerID="c5369d27d78341b60989b6d5bc677b59af9e5b0e210806b6b6b392838f1c1bad"
Jul 27 21:30:23.304885 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.304866    2580 scope.go:117] "RemoveContainer" containerID="5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"
Jul 27 21:30:23.311549 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.311520    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:30:23.314672 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.314569    2580 scope.go:117] "RemoveContainer" containerID="5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"
Jul 27 21:30:23.315007 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:30:23.314973    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7\": container with ID starting with 5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7 not found: ID does not exist" containerID="5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"
Jul 27 21:30:23.315094 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.315007    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7"} err="failed to get container status \"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7\": rpc error: code = NotFound desc = could not find container \"5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7\": container with ID starting with 5136cb84f5401a57af23ff3371032518395dcaadb79714a1bdb81cc5b5c217b7 not found: ID does not exist"
Jul 27 21:30:23.317135 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.317113    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9d44e2-7494d4849b-xvzzv"]
Jul 27 21:30:23.328697 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.328666    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:30:23.332098 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.332071    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9d44e2-59d5b775b4-5zxm4"]
Jul 27 21:30:23.344671 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.344643    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:30:23.351238 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:23.351197    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9d44e2-5c5b575964-vhprz"]
Jul 27 21:30:24.806015 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:24.805941    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" path="/var/lib/kubelet/pods/88e85b0b-dfef-4354-9deb-1e4795e9b33a/volumes"
Jul 27 21:30:24.806395 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:24.806319    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb0f98c-e26c-4154-b8d4-ddea93c13562" path="/var/lib/kubelet/pods/adb0f98c-e26c-4154-b8d4-ddea93c13562/volumes"
Jul 27 21:30:24.806642 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:24.806629    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" path="/var/lib/kubelet/pods/e9d59c9f-c06e-4789-b523-e1a01cf83bba/volumes"
Jul 27 21:30:25.905820 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:25.905781    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:26.313784 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:26.313745    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.255017 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:32.254965    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.357355 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:32.357312    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.544122 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:32.544026    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.753141 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:32.753102    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:48.319146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:48.319105    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:48.386095 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:48.386061    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:48.487235 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:48.487181    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:52.944939 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:52.944889    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:53.366976 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:53.366940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:59.129596 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:59.129547    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:59.230069 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:59.230029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:59.413866 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:59.413768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:59.668583 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:30:59.668480    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:00.015651 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:00.015610    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:00.345231 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:00.345120    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:00.476092 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:00.476049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.558408 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.558369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.634907 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.634869    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.636613 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.636593    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:31:03.636975 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.636963    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" containerName="payload-pre-processing"
Jul 27 21:31:03.637017 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.636978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" containerName="payload-pre-processing"
Jul 27 21:31:03.637056 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637017    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb0f98c-e26c-4154-b8d4-ddea93c13562" containerName="payload-processing"
Jul 27 21:31:03.637056 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637025    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb0f98c-e26c-4154-b8d4-ddea93c13562" containerName="payload-processing"
Jul 27 21:31:03.637056 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637050    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" containerName="maas-api"
Jul 27 21:31:03.637148 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637057    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" containerName="maas-api"
Jul 27 21:31:03.637182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637150    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e85b0b-dfef-4354-9deb-1e4795e9b33a" containerName="maas-api"
Jul 27 21:31:03.637182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb0f98c-e26c-4154-b8d4-ddea93c13562" containerName="payload-processing"
Jul 27 21:31:03.637182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.637166    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d59c9f-c06e-4789-b523-e1a01cf83bba" containerName="payload-pre-processing"
Jul 27 21:31:03.640630 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.640606    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.643373 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.643343    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-6dd721-serving-cert\""
Jul 27 21:31:03.652384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.652357    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:31:03.708809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.708770    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.713977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.713946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.714127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.713992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.714178 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.714121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9kj\" (UniqueName: \"kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.814853 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.814754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.814853 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.814800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.815059 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.814878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9kj\" (UniqueName: \"kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.815529 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.815506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.817519 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.817496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.824310 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.824287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9kj\" (UniqueName: \"kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj\") pod \"maas-api-e2e-shared-a-6dd721-74679586b9-cctnh\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:03.842452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.842413    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.953949 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:03.953908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:04.104594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.104476    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:31:04.134332 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.134294    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:04.232059 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.232021    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:04.264026 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.263992    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:31:04.268998 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.268962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.276950 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.276920    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-6dd721-dockercfg-h78q6\""
Jul 27 21:31:04.277185 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.277149    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-6dd721\""
Jul 27 21:31:04.279446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.279422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:31:04.319452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.319395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kvk\" (UniqueName: \"kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.319628 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.319514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.420700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.420606    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.420856 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.420717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kvk\" (UniqueName: \"kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.421440 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.421418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.430694 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.430669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kvk\" (UniqueName: \"kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk\") pod \"payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.494540 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.494507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" event={"ID":"c4719105-3444-4894-be97-1e3f133309f4","Type":"ContainerStarted","Data":"d8be0b7851869e63e0363aee48c312a5742f3122d3a23af2de2e7974e7211698"}
Jul 27 21:31:04.581187 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.581153    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:04.744924 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:04.744891    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:31:04.748135 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:31:04.748091    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2f38d2_394d_4e24_8968_888489dcabe8.slice/crio-102215759eeb13b5c6b6aa75c5687bfd166e388a578d549b772dc65d02aa2f91 WatchSource:0}: Error finding container 102215759eeb13b5c6b6aa75c5687bfd166e388a578d549b772dc65d02aa2f91: Status 404 returned error can't find the container with id 102215759eeb13b5c6b6aa75c5687bfd166e388a578d549b772dc65d02aa2f91
Jul 27 21:31:05.500527 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.500477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" event={"ID":"c4719105-3444-4894-be97-1e3f133309f4","Type":"ContainerStarted","Data":"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"}
Jul 27 21:31:05.500745 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.500620    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:05.502116 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.502082    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" event={"ID":"4c2f38d2-394d-4e24-8968-888489dcabe8","Type":"ContainerStarted","Data":"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"}
Jul 27 21:31:05.502285 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.502132    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" event={"ID":"4c2f38d2-394d-4e24-8968-888489dcabe8","Type":"ContainerStarted","Data":"102215759eeb13b5c6b6aa75c5687bfd166e388a578d549b772dc65d02aa2f91"}
Jul 27 21:31:05.502285 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.502195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:05.521721 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.521653    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" podStartSLOduration=2.132923804 podStartE2EDuration="2.521633314s" totalImagesPullingTime="388.70951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:05.52118635 +0000 UTC m=+2371.295118767" watchObservedRunningTime="2026-07-27 21:31:05.521633314 +0000 UTC m=+2371.295565736"
Jul 27 21:31:05.541412 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:05.541360    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" podStartSLOduration=1.541342914 podStartE2EDuration="1.541342914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:05.540414919 +0000 UTC m=+2371.314347342" watchObservedRunningTime="2026-07-27 21:31:05.541342914 +0000 UTC m=+2371.315275379"
Jul 27 21:31:08.079528 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:08.079493    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:08.409609 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:08.409484    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:08.736048 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:08.736004    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:08.792427 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:08.792388    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:11.512519 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.512489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:31:11.613586 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.613546    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:11.701621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.701580    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:11.704500 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.704470    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:31:11.708891 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.708871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.711396 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.711373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5f50cb-serving-cert\""
Jul 27 21:31:11.718169 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.718125    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:31:11.773332 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.773237    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:11.789299 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.789257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjzh\" (UniqueName: \"kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.789453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.789324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.789453 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.789389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.872961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.872930    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:11.890809 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.890773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.890957 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.890838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.890957 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.890927    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjzh\" (UniqueName: \"kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.891611 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.891557    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.893478 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.893451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:11.899602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:11.899578    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjzh\" (UniqueName: \"kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh\") pod \"maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:12.022068 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.022025    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:12.168539 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.168509    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:31:12.170813 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:31:12.170780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590072cb_2465_48d2_a59e_324390e51fd0.slice/crio-743d8b1fb6c682031fea5a9e488b66d69bb9e1f49d0862fe120d4427a346d0d6 WatchSource:0}: Error finding container 743d8b1fb6c682031fea5a9e488b66d69bb9e1f49d0862fe120d4427a346d0d6: Status 404 returned error can't find the container with id 743d8b1fb6c682031fea5a9e488b66d69bb9e1f49d0862fe120d4427a346d0d6
Jul 27 21:31:12.294817 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.294717    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:12.392104 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.392068    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:12.412412 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.412376    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:31:12.416992 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.416971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.419792 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.419746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-5f50cb\""
Jul 27 21:31:12.419925 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.419786    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-5f50cb-dockercfg-zkc7f\""
Jul 27 21:31:12.428008 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.427975    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:31:12.496179 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.496150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtl7b\" (UniqueName: \"kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.496405 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.496187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.537365 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.537321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" event={"ID":"590072cb-2465-48d2-a59e-324390e51fd0","Type":"ContainerStarted","Data":"743d8b1fb6c682031fea5a9e488b66d69bb9e1f49d0862fe120d4427a346d0d6"}
Jul 27 21:31:12.597604 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.597574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl7b\" (UniqueName: \"kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.597749 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.597614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.598337 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.598316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.606632 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.606600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl7b\" (UniqueName: \"kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b\") pod \"payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.729246 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.729175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:12.882368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:12.882335    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:31:12.884107 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:31:12.884077    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f220613_d4de_4196_8bc3_6a91db4f7b98.slice/crio-0b384484ef5beb59c69fe0de8fdf6fba3b7691dd7138565b9eac1860e9a0a86d WatchSource:0}: Error finding container 0b384484ef5beb59c69fe0de8fdf6fba3b7691dd7138565b9eac1860e9a0a86d: Status 404 returned error can't find the container with id 0b384484ef5beb59c69fe0de8fdf6fba3b7691dd7138565b9eac1860e9a0a86d
Jul 27 21:31:13.544734 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.544695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" event={"ID":"590072cb-2465-48d2-a59e-324390e51fd0","Type":"ContainerStarted","Data":"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"}
Jul 27 21:31:13.545260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.544789    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:13.546387 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.546365    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" event={"ID":"6f220613-d4de-4196-8bc3-6a91db4f7b98","Type":"ContainerStarted","Data":"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"}
Jul 27 21:31:13.546506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.546402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" event={"ID":"6f220613-d4de-4196-8bc3-6a91db4f7b98","Type":"ContainerStarted","Data":"0b384484ef5beb59c69fe0de8fdf6fba3b7691dd7138565b9eac1860e9a0a86d"}
Jul 27 21:31:13.546506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.546439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:13.563785 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.563725    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" podStartSLOduration=2.160153449 podStartE2EDuration="2.563709028s" totalImagesPullingTime="403.555579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:13.562706132 +0000 UTC m=+2379.336638550" watchObservedRunningTime="2026-07-27 21:31:13.563709028 +0000 UTC m=+2379.337641449"
Jul 27 21:31:13.579499 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:13.579439    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" podStartSLOduration=1.579421572 podStartE2EDuration="1.579421572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:13.577880852 +0000 UTC m=+2379.351813277" watchObservedRunningTime="2026-07-27 21:31:13.579421572 +0000 UTC m=+2379.353353991"
Jul 27 21:31:16.250155 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:16.250113    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:16.509578 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:16.509493    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:31:19.556848 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:19.556822    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:31:24.554719 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:24.554684    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:31:28.826808 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:28.826773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:35.750594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:35.750559    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:31:35.757101 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:35.757076    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:31:55.400371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.400277    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:31:55.405086 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.405063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.407902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.407878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"openshift-service-ca.crt\""
Jul 27 21:31:55.408036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.407924    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"kube-root-ca.crt\""
Jul 27 21:31:55.409303 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.409137    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"default-dockercfg-54kfm\""
Jul 27 21:31:55.409303 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.409204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"auth-test-model-5f50cb-kserve-self-signed-certs\""
Jul 27 21:31:55.412519 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.412493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:31:55.435993 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.435961    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:55.509687 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509646    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bvb\" (UniqueName: \"kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.509879 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.509879 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.509879 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.509879 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.510046 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.509895    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.610746 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.610746 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bvb\" (UniqueName: \"kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611127 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.610873    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611360 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.611273    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611360 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.611293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.611431 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.611354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.613180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.613158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.613442 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.613426    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.619263 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.619241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bvb\" (UniqueName: \"kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb\") pod \"auth-test-model-5f50cb-kserve-5f86595495-tzn2w\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") " pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.717469 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.717426    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:31:55.861746 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:55.861722    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:31:55.864002 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:31:55.863972    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d7ce8c_76ea_452e_94b4_5adcd56ac2bb.slice/crio-292049c50e90efc342922409304afe8d68a9bf9a31bb48314138765ecdcd4b92 WatchSource:0}: Error finding container 292049c50e90efc342922409304afe8d68a9bf9a31bb48314138765ecdcd4b92: Status 404 returned error can't find the container with id 292049c50e90efc342922409304afe8d68a9bf9a31bb48314138765ecdcd4b92
Jul 27 21:31:56.776700 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:56.776651    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" event={"ID":"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb","Type":"ContainerStarted","Data":"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"}
Jul 27 21:31:56.777176 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:56.776725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" event={"ID":"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb","Type":"ContainerStarted","Data":"292049c50e90efc342922409304afe8d68a9bf9a31bb48314138765ecdcd4b92"}
Jul 27 21:31:56.794272 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:56.794202    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" podStartSLOduration=1.794185516 podStartE2EDuration="1.794185516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:56.793831072 +0000 UTC m=+2422.567763492" watchObservedRunningTime="2026-07-27 21:31:56.794185516 +0000 UTC m=+2422.568117937"
Jul 27 21:31:58.382772 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:31:58.382739    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:05.718070 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:32:05.718026    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:32:05.718499 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:32:05.718083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:32:05.730804 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:32:05.730777    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:32:05.837765 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:32:05.837729    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:33:50.938642 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:33:50.938599    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:33:51.010407 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:33:51.010375    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:15.370445 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:34:15.370409    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:35.829371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:34:35.829323    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:35:04.932426 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:35:04.932334    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:01.549547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:01.549503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:11.190865 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:11.190826    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:12.077371 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:12.077333    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:36:12.104820 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:12.104787    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:21.502471 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.502435    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:36:21.506418 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.506392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.510082 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.510040    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"kube-root-ca.crt\""
Jul 27 21:36:21.510321 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.510041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"openshift-service-ca.crt\""
Jul 27 21:36:21.510321 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.510041    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"sub-test-model-6dd721-kserve-self-signed-certs\""
Jul 27 21:36:21.510321 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.510042    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"default-dockercfg-7jnzq\""
Jul 27 21:36:21.516133 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.516099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:36:21.535234 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.535176    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:21.557644 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fqvx\" (UniqueName: \"kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.557812 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.557812 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.557812 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.557935 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.557935 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.557883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fqvx\" (UniqueName: \"kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.658980 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.658958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.659071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.659003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.659071 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.659032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.661192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.661162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.661376 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.661357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.668013 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.667984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fqvx\" (UniqueName: \"kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx\") pod \"sub-test-model-6dd721-kserve-5c879958f9-zwzv4\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") " pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.818364 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.818267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:21.981555 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:21.981475    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:36:21.987117 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:36:21.987082    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc734a6_5b29_40e9_a865_e2e247281625.slice/crio-90f50acf39b1ff5876ec808c1aeab02f98ae454b9a427fa767fec1b8d1d56855 WatchSource:0}: Error finding container 90f50acf39b1ff5876ec808c1aeab02f98ae454b9a427fa767fec1b8d1d56855: Status 404 returned error can't find the container with id 90f50acf39b1ff5876ec808c1aeab02f98ae454b9a427fa767fec1b8d1d56855
Jul 27 21:36:22.113442 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:22.113356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" event={"ID":"8fc734a6-5b29-40e9-a865-e2e247281625","Type":"ContainerStarted","Data":"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"}
Jul 27 21:36:22.113442 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:22.113399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" event={"ID":"8fc734a6-5b29-40e9-a865-e2e247281625","Type":"ContainerStarted","Data":"90f50acf39b1ff5876ec808c1aeab02f98ae454b9a427fa767fec1b8d1d56855"}
Jul 27 21:36:22.138856 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:22.138788    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" podStartSLOduration=1.138767432 podStartE2EDuration="1.138767432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:22.135705453 +0000 UTC m=+2687.909637873" watchObservedRunningTime="2026-07-27 21:36:22.138767432 +0000 UTC m=+2687.912699909"
Jul 27 21:36:27.098632 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.098539    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" podUID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" containerName="main" containerID="cri-o://9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc" gracePeriod=45
Jul 27 21:36:27.351728 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.351664    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:36:27.413622 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2bvb\" (UniqueName: \"kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.413622 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413630    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.413846 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413688    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.413846 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413728    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.413846 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.414006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.413848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir\") pod \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\" (UID: \"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb\") "
Jul 27 21:36:27.414060 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.414013    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home" (OuterVolumeSpecName: "home") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:27.414060 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.414051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:27.414181 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.414165    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.414252 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.414169    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:27.414252 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.414184    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.416091 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.416067    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb" (OuterVolumeSpecName: "kube-api-access-z2bvb") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "kube-api-access-z2bvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:27.416393 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.416372    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:27.416393 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.416377    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm" (OuterVolumeSpecName: "dshm") pod "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" (UID: "97d7ce8c-76ea-452e-94b4-5adcd56ac2bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:27.514831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.514788    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.514831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.514821    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.514831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.514831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bvb\" (UniqueName: \"kubernetes.io/projected/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-kube-api-access-z2bvb\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.514831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:27.514840    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:36:28.147293 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.147259    2580 generic.go:356] "Generic (PLEG): container finished" podID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" containerID="9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc" exitCode=0
Jul 27 21:36:28.147760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.147306    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" event={"ID":"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb","Type":"ContainerDied","Data":"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"}
Jul 27 21:36:28.147760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.147309    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"
Jul 27 21:36:28.147760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.147332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w" event={"ID":"97d7ce8c-76ea-452e-94b4-5adcd56ac2bb","Type":"ContainerDied","Data":"292049c50e90efc342922409304afe8d68a9bf9a31bb48314138765ecdcd4b92"}
Jul 27 21:36:28.147760 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.147348    2580 scope.go:117] "RemoveContainer" containerID="9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"
Jul 27 21:36:28.157188 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.157170    2580 scope.go:117] "RemoveContainer" containerID="9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"
Jul 27 21:36:28.157457 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:36:28.157440    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc\": container with ID starting with 9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc not found: ID does not exist" containerID="9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"
Jul 27 21:36:28.157506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.157465    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc"} err="failed to get container status \"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc\": rpc error: code = NotFound desc = could not find container \"9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc\": container with ID starting with 9762e3ff50d5d27e69cf50d44585e3a76739458103df79e1638aace5b677ebfc not found: ID does not exist"
Jul 27 21:36:28.169251 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.169208    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:36:28.173673 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.173652    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/auth-test-model-5f50cb-kserve-5f86595495-tzn2w"]
Jul 27 21:36:28.806569 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:28.806535    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" path="/var/lib/kubelet/pods/97d7ce8c-76ea-452e-94b4-5adcd56ac2bb/volumes"
Jul 27 21:36:31.818901 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:31.818847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:31.819384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:31.819162    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:31.831841 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:31.831814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:32.179902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:32.179810    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:36:35.781473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:35.781430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:36:35.789334 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:35.789306    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:36:39.621431 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:36:39.621389    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:56.140678 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:37:56.140587    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:37:56.170151 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:37:56.170111    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:57.010395 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:37:57.010325    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:08.202711 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.202672    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:38:08.203317 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.203295    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" containerName="main"
Jul 27 21:38:08.203399 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.203319    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" containerName="main"
Jul 27 21:38:08.203474 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.203461    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d7ce8c-76ea-452e-94b4-5adcd56ac2bb" containerName="main"
Jul 27 21:38:08.206485 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.206462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.209557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.209533    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"rate-test-model-6dd721-kserve-self-signed-certs\""
Jul 27 21:38:08.219573 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.219547    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:38:08.242498 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.242434    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:08.260397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.260581 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fkb\" (UniqueName: \"kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.260581 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.260699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.260699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.260699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.260670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.361821 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.361778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.361821 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.361827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.361862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.361937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.361990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fkb\" (UniqueName: \"kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.362018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362346 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.362282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362346 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.362331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.362462 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.362428    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.364448 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.364432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.364597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.364582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.371084 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.371057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fkb\" (UniqueName: \"kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb\") pod \"rate-test-model-6dd721-kserve-65f49d4947-v84tp\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") " pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.519708 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.519666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:08.668126 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:08.668104    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:38:08.674285 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:38:08.674245    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a57dd81_8f09_4cde_a36b_6b551754e79c.slice/crio-5c760778dff0d5ab41f4b1cdcfee4c8348832bb0b88cdcbefaf75975cefd741b WatchSource:0}: Error finding container 5c760778dff0d5ab41f4b1cdcfee4c8348832bb0b88cdcbefaf75975cefd741b: Status 404 returned error can't find the container with id 5c760778dff0d5ab41f4b1cdcfee4c8348832bb0b88cdcbefaf75975cefd741b
Jul 27 21:38:09.659137 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:09.659099    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" event={"ID":"4a57dd81-8f09-4cde-a36b-6b551754e79c","Type":"ContainerStarted","Data":"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"}
Jul 27 21:38:09.659551 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:09.659145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" event={"ID":"4a57dd81-8f09-4cde-a36b-6b551754e79c","Type":"ContainerStarted","Data":"5c760778dff0d5ab41f4b1cdcfee4c8348832bb0b88cdcbefaf75975cefd741b"}
Jul 27 21:38:09.678900 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:09.678841    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" podStartSLOduration=1.678824166 podStartE2EDuration="1.678824166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:09.677735852 +0000 UTC m=+2795.451668272" watchObservedRunningTime="2026-07-27 21:38:09.678824166 +0000 UTC m=+2795.452756618"
Jul 27 21:38:11.163023 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.162973    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" podUID="8fc734a6-5b29-40e9-a865-e2e247281625" containerName="main" containerID="cri-o://6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1" gracePeriod=45
Jul 27 21:38:11.432375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.432346    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:38:11.493805 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fqvx\" (UniqueName: \"kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.493976 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.493976 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.493976 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.493976 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.494257 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.493978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs\") pod \"8fc734a6-5b29-40e9-a865-e2e247281625\" (UID: \"8fc734a6-5b29-40e9-a865-e2e247281625\") "
Jul 27 21:38:11.494257 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.494147    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:11.494257 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.494186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home" (OuterVolumeSpecName: "home") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:11.494409 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.494265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:11.494409 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.494330    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.494409 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.494349    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.496231 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.496194    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx" (OuterVolumeSpecName: "kube-api-access-7fqvx") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "kube-api-access-7fqvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:38:11.496307 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.496258    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm" (OuterVolumeSpecName: "dshm") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:11.496397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.496379    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc734a6-5b29-40e9-a865-e2e247281625" (UID: "8fc734a6-5b29-40e9-a865-e2e247281625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:38:11.595368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.595332    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.595368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.595362    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc734a6-5b29-40e9-a865-e2e247281625-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.595368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.595373    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc734a6-5b29-40e9-a865-e2e247281625-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.595368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.595383    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fqvx\" (UniqueName: \"kubernetes.io/projected/8fc734a6-5b29-40e9-a865-e2e247281625-kube-api-access-7fqvx\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:38:11.670162 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.670072    2580 generic.go:356] "Generic (PLEG): container finished" podID="8fc734a6-5b29-40e9-a865-e2e247281625" containerID="6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1" exitCode=0
Jul 27 21:38:11.670162 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.670118    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"
Jul 27 21:38:11.670375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.670155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" event={"ID":"8fc734a6-5b29-40e9-a865-e2e247281625","Type":"ContainerDied","Data":"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"}
Jul 27 21:38:11.670375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.670201    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4" event={"ID":"8fc734a6-5b29-40e9-a865-e2e247281625","Type":"ContainerDied","Data":"90f50acf39b1ff5876ec808c1aeab02f98ae454b9a427fa767fec1b8d1d56855"}
Jul 27 21:38:11.670375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.670239    2580 scope.go:117] "RemoveContainer" containerID="6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"
Jul 27 21:38:11.680318 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.680294    2580 scope.go:117] "RemoveContainer" containerID="6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"
Jul 27 21:38:11.680641 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:38:11.680622    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1\": container with ID starting with 6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1 not found: ID does not exist" containerID="6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"
Jul 27 21:38:11.680699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.680651    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1"} err="failed to get container status \"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1\": rpc error: code = NotFound desc = could not find container \"6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1\": container with ID starting with 6d020a09d6b477ba78a574010de0bc05230f86cca1fd5adec61729f475273bf1 not found: ID does not exist"
Jul 27 21:38:11.690959 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.690930    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:38:11.694758 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:11.694731    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/sub-test-model-6dd721-kserve-5c879958f9-zwzv4"]
Jul 27 21:38:12.805827 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:12.805787    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc734a6-5b29-40e9-a865-e2e247281625" path="/var/lib/kubelet/pods/8fc734a6-5b29-40e9-a865-e2e247281625/volumes"
Jul 27 21:38:15.776542 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:15.776503    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:18.519953 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:18.519917    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:18.519953 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:18.519963    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:18.532988 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:18.532956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:18.719538 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:18.719496    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:38:34.834109 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:38:34.834063    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:58.638631 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:39:58.638595    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:39:58.653925 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:39:58.653890    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:59.830415 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:39:59.830378    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:13.658160 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.658107    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" podUID="4a57dd81-8f09-4cde-a36b-6b551754e79c" containerName="main" containerID="cri-o://68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df" gracePeriod=45
Jul 27 21:40:13.916568 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.916499    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:40:13.937692 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.937857 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.937857 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.937857 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937770    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.937985 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5fkb\" (UniqueName: \"kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.937985 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.937924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache\") pod \"4a57dd81-8f09-4cde-a36b-6b551754e79c\" (UID: \"4a57dd81-8f09-4cde-a36b-6b551754e79c\") "
Jul 27 21:40:13.938251 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.938203    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:13.938251 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.938235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home" (OuterVolumeSpecName: "home") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:13.938446 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.938421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:13.940228 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.940190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb" (OuterVolumeSpecName: "kube-api-access-d5fkb") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "kube-api-access-d5fkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:40:13.940577 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.940550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm" (OuterVolumeSpecName: "dshm") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:13.940699 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:13.940673    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a57dd81-8f09-4cde-a36b-6b551754e79c" (UID: "4a57dd81-8f09-4cde-a36b-6b551754e79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:40:14.038675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038639    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fkb\" (UniqueName: \"kubernetes.io/projected/4a57dd81-8f09-4cde-a36b-6b551754e79c-kube-api-access-d5fkb\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.038675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038668    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.038675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038679    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.038675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038688    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a57dd81-8f09-4cde-a36b-6b551754e79c-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.038952 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038698    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.038952 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.038706    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a57dd81-8f09-4cde-a36b-6b551754e79c-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.297336 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.297298    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a57dd81-8f09-4cde-a36b-6b551754e79c" containerID="68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df" exitCode=0
Jul 27 21:40:14.297523 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.297343    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"
Jul 27 21:40:14.297523 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.297379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" event={"ID":"4a57dd81-8f09-4cde-a36b-6b551754e79c","Type":"ContainerDied","Data":"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"}
Jul 27 21:40:14.297523 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.297425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp" event={"ID":"4a57dd81-8f09-4cde-a36b-6b551754e79c","Type":"ContainerDied","Data":"5c760778dff0d5ab41f4b1cdcfee4c8348832bb0b88cdcbefaf75975cefd741b"}
Jul 27 21:40:14.297523 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.297446    2580 scope.go:117] "RemoveContainer" containerID="68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"
Jul 27 21:40:14.308894 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.308876    2580 scope.go:117] "RemoveContainer" containerID="68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"
Jul 27 21:40:14.309173 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:40:14.309147    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df\": container with ID starting with 68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df not found: ID does not exist" containerID="68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"
Jul 27 21:40:14.309250 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.309182    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df"} err="failed to get container status \"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df\": rpc error: code = NotFound desc = could not find container \"68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df\": container with ID starting with 68429bb10f25f9a228667bee78ab0afa556cb2db05a287687acb9303172ad1df not found: ID does not exist"
Jul 27 21:40:14.321697 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.321669    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:40:14.325913 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.325891    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/rate-test-model-6dd721-kserve-65f49d4947-v84tp"]
Jul 27 21:40:14.805003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:14.804973    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a57dd81-8f09-4cde-a36b-6b551754e79c" path="/var/lib/kubelet/pods/4a57dd81-8f09-4cde-a36b-6b551754e79c/volumes"
Jul 27 21:40:18.951039 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:18.950988    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:19.260385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:19.260331    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:19.646568 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:19.646471    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:22.362950 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:22.362899    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:22.459469 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:22.459432    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:22.634189 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:22.634097    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:22.734966 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:22.734933    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:23.319010 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.318973    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:23.419395 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.419360    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:23.453226 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:40:23.453607 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453595    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc734a6-5b29-40e9-a865-e2e247281625" containerName="main"
Jul 27 21:40:23.453654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453609    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc734a6-5b29-40e9-a865-e2e247281625" containerName="main"
Jul 27 21:40:23.453654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a57dd81-8f09-4cde-a36b-6b551754e79c" containerName="main"
Jul 27 21:40:23.453654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453628    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a57dd81-8f09-4cde-a36b-6b551754e79c" containerName="main"
Jul 27 21:40:23.453751 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453701    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a57dd81-8f09-4cde-a36b-6b551754e79c" containerName="main"
Jul 27 21:40:23.453751 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.453713    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc734a6-5b29-40e9-a865-e2e247281625" containerName="main"
Jul 27 21:40:23.458645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.458615    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.463241 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.461753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b4bf5c-dockercfg-mh75g\""
Jul 27 21:40:23.463241 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.461790    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b4bf5c\""
Jul 27 21:40:23.469908 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.469878    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:40:23.523835 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.523790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.524035 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.523850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxd88\" (UniqueName: \"kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.625159 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.625072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.625159 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.625131    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxd88\" (UniqueName: \"kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.625918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.625897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.633938 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.633912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxd88\" (UniqueName: \"kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88\") pod \"payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.777768 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.777723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:23.936153 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:23.936120    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:40:23.938504 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:40:23.938472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f14f63_b7cb_4971_9fc9_f659d3de9d68.slice/crio-abe455e2f8fc250e95f531afb3b8ad866346bff571ba012fb405ec15236bb728 WatchSource:0}: Error finding container abe455e2f8fc250e95f531afb3b8ad866346bff571ba012fb405ec15236bb728: Status 404 returned error can't find the container with id abe455e2f8fc250e95f531afb3b8ad866346bff571ba012fb405ec15236bb728
Jul 27 21:40:24.351884 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:24.351843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" event={"ID":"18f14f63-b7cb-4971-9fc9-f659d3de9d68","Type":"ContainerStarted","Data":"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"}
Jul 27 21:40:24.352067 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:24.351900    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" event={"ID":"18f14f63-b7cb-4971-9fc9-f659d3de9d68","Type":"ContainerStarted","Data":"abe455e2f8fc250e95f531afb3b8ad866346bff571ba012fb405ec15236bb728"}
Jul 27 21:40:24.352067 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:24.351923    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:24.368063 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:24.368014    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" podStartSLOduration=1.367998589 podStartE2EDuration="1.367998589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:24.367307721 +0000 UTC m=+2930.141240142" watchObservedRunningTime="2026-07-27 21:40:24.367998589 +0000 UTC m=+2930.141931009"
Jul 27 21:40:26.905659 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:26.905609    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:35.359049 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:35.359016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:40:37.898375 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:37.898336    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:38.240073 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:38.240041    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:38.457373 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:38.457335    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:41.459762 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.459722    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:41.559693 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.559662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:40:41.562729 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.562687    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:41.564602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.564583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.567736 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.567715    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9e51a0-serving-cert\""
Jul 27 21:40:41.580374 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.580350    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:40:41.650082 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.650034    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:41.702286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.702254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.702286 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.702289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.702544 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.702376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclb8\" (UniqueName: \"kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.759661 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.759569    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:41.803688 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.803658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.803688 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.803688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.803851 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.803730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclb8\" (UniqueName: \"kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.804400 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.804376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.806140 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.806114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.812644 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.812622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclb8\" (UniqueName: \"kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8\") pod \"maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:41.875900 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:41.875861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:42.019655 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.019627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:40:42.021879 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:40:42.021852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb52d9c9_27f4_42b7_8254_cd2cecf81984.slice/crio-8039e1301989700527e1a08e7ac47039ac307127dfafd4084f04c5a3a9784107 WatchSource:0}: Error finding container 8039e1301989700527e1a08e7ac47039ac307127dfafd4084f04c5a3a9784107: Status 404 returned error can't find the container with id 8039e1301989700527e1a08e7ac47039ac307127dfafd4084f04c5a3a9784107
Jul 27 21:40:42.023524 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.023504    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:40:42.147457 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.147419    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:42.244652 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.244619    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:42.273090 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.273016    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:40:42.278123 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.278103    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.281186 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.281162    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9e51a0-dockercfg-ks5dq\""
Jul 27 21:40:42.281804 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.281789    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9e51a0\""
Jul 27 21:40:42.289140 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.289114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:40:42.409891 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.409859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8jz\" (UniqueName: \"kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.410036 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.409979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.444002 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.443969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" event={"ID":"cb52d9c9-27f4-42b7-8254-cd2cecf81984","Type":"ContainerStarted","Data":"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"}
Jul 27 21:40:42.444180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.444022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" event={"ID":"cb52d9c9-27f4-42b7-8254-cd2cecf81984","Type":"ContainerStarted","Data":"8039e1301989700527e1a08e7ac47039ac307127dfafd4084f04c5a3a9784107"}
Jul 27 21:40:42.444180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.444067    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:42.462405 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.462350    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" podStartSLOduration=1.143592238 podStartE2EDuration="1.462334609s" totalImagesPullingTime="318.742371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:42.460889256 +0000 UTC m=+2948.234821678" watchObservedRunningTime="2026-07-27 21:40:42.462334609 +0000 UTC m=+2948.236267029"
Jul 27 21:40:42.511443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.511399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8jz\" (UniqueName: \"kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.511662 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.511487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.512276 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.512256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.523655 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.523597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8jz\" (UniqueName: \"kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz\") pod \"payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.590557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.590512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:42.950006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:42.949976    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:40:42.952146 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:40:42.952115    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c421612_18c9_452b_b911_46e6a124bed1.slice/crio-7d35e9c51b45c0426c1965fa02d293f4553aaf6a45033320e6646475e54eab7c WatchSource:0}: Error finding container 7d35e9c51b45c0426c1965fa02d293f4553aaf6a45033320e6646475e54eab7c: Status 404 returned error can't find the container with id 7d35e9c51b45c0426c1965fa02d293f4553aaf6a45033320e6646475e54eab7c
Jul 27 21:40:43.451385 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:43.451335    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" event={"ID":"7c421612-18c9-452b-b911-46e6a124bed1","Type":"ContainerStarted","Data":"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"}
Jul 27 21:40:43.451582 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:43.451398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" event={"ID":"7c421612-18c9-452b-b911-46e6a124bed1","Type":"ContainerStarted","Data":"7d35e9c51b45c0426c1965fa02d293f4553aaf6a45033320e6646475e54eab7c"}
Jul 27 21:40:43.451582 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:43.451536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:40:43.470868 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:43.470818    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" podStartSLOduration=1.47078516 podStartE2EDuration="1.47078516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:43.47041545 +0000 UTC m=+2949.244347870" watchObservedRunningTime="2026-07-27 21:40:43.47078516 +0000 UTC m=+2949.244717582"
Jul 27 21:40:45.996137 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:45.996087    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:48.456764 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:48.456735    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:40:54.459120 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:40:54.459086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:41:03.540608 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:03.540573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:41:03.677775 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:03.677740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq_7c421612-18c9-452b-b911-46e6a124bed1/payload-processing/0.log"
Jul 27 21:41:14.204372 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.204330    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:41:14.208363 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.208328    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.211961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.211888    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4bf5c\"/\"ipp-route-b4bf5c-kserve-self-signed-certs\""
Jul 27 21:41:14.211961 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.211921    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4bf5c\"/\"openshift-service-ca.crt\""
Jul 27 21:41:14.212179 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.211963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b4bf5c\"/\"default-dockercfg-jtgj5\""
Jul 27 21:41:14.212179 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.211999    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b4bf5c\"/\"kube-root-ca.crt\""
Jul 27 21:41:14.220310 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.220283    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:41:14.254158 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.254124    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:14.316594 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.316778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.316778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdd8h\" (UniqueName: \"kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.316778 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.316958 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316789    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.316958 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.316838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417620 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdd8h\" (UniqueName: \"kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.417815 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.417761    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.418084 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.418005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.418149 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.418094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.418149 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.418131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.420133 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.420101    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.420473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.420456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.427683 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.427661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdd8h\" (UniqueName: \"kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h\") pod \"ipp-route-b4bf5c-kserve-6455c974b7-bx25d\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") " pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.519311 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.519273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:14.668653 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:14.668623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:41:14.670335 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:41:14.670311    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64965432_3f06_4c60_a612_cd08f5016649.slice/crio-46a90b20e1ca59ea11b003a2e8d3f093058286b132429cedff1893a55a60e4f9 WatchSource:0}: Error finding container 46a90b20e1ca59ea11b003a2e8d3f093058286b132429cedff1893a55a60e4f9: Status 404 returned error can't find the container with id 46a90b20e1ca59ea11b003a2e8d3f093058286b132429cedff1893a55a60e4f9
Jul 27 21:41:15.619829 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:15.619788    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" event={"ID":"64965432-3f06-4c60-a612-cd08f5016649","Type":"ContainerStarted","Data":"ef3cce4861d0d19d3379db57532606128f1b3e3f722d2d55c1a09260eb052e15"}
Jul 27 21:41:15.619829 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:15.619833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" event={"ID":"64965432-3f06-4c60-a612-cd08f5016649","Type":"ContainerStarted","Data":"46a90b20e1ca59ea11b003a2e8d3f093058286b132429cedff1893a55a60e4f9"}
Jul 27 21:41:15.640272 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:15.640187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" podStartSLOduration=1.640169478 podStartE2EDuration="1.640169478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:15.639044246 +0000 UTC m=+2981.412976679" watchObservedRunningTime="2026-07-27 21:41:15.640169478 +0000 UTC m=+2981.414101898"
Jul 27 21:41:24.519535 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:24.519502    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:24.520012 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:24.519706    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:24.532331 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:24.532304    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:24.676452 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:24.676421    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:41:35.812776 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:35.812744    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:41:35.821397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:35.821379    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xk7hl_b1992c52-3136-473a-80ce-bee25c04104a/ovn-acl-logging/0.log"
Jul 27 21:41:56.231603 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:56.231572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:41:56.361942 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:56.361913    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq_7c421612-18c9-452b-b911-46e6a124bed1/payload-processing/0.log"
Jul 27 21:41:56.655030 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:56.654918    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:56.997733 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:56.997702    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:57.669878 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:57.669840    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:59.813001 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.812954    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:59.912455 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.912417    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:59.914869 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.914835    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:41:59.919578 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.919552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:41:59.922798 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.922775    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-400a5b-serving-cert\""
Jul 27 21:41:59.931515 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:41:59.931485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:42:00.006364 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.006305    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:00.055922 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.055880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv2t6\" (UniqueName: \"kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.055922 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.055923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.056148 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.056019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.157287 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.157170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv2t6\" (UniqueName: \"kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.157287 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.157209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.157547 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.157329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.157547 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:42:00.157460    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-serving-cert: secret "maas-api-e2e-ipp-cleanup-400a5b-serving-cert" not found
Jul 27 21:42:00.157547 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:42:00.157519    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls podName:29e52440-5691-451a-b402-8eb5af4a9793 nodeName:}" failed. No retries permitted until 2026-07-27 21:42:00.657504206 +0000 UTC m=+3026.431436604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" (UID: "29e52440-5691-451a-b402-8eb5af4a9793") : secret "maas-api-e2e-ipp-cleanup-400a5b-serving-cert" not found
Jul 27 21:42:00.158014 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.157990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.166060 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.166030    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv2t6\" (UniqueName: \"kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.388109 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.388079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:00.649164 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.649128    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:00.662586 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.662550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.665864 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.665825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.747675 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.747632    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:00.831673 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.831641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:00.985089 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:00.985065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:42:00.987804 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:42:00.987774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e52440_5691_451a_b402_8eb5af4a9793.slice/crio-78ffc389df11230bb61fb348ce36311a1c4eabe2e564d07af22f9e7387ef6808 WatchSource:0}: Error finding container 78ffc389df11230bb61fb348ce36311a1c4eabe2e564d07af22f9e7387ef6808: Status 404 returned error can't find the container with id 78ffc389df11230bb61fb348ce36311a1c4eabe2e564d07af22f9e7387ef6808
Jul 27 21:42:01.864618 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:01.864575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" event={"ID":"29e52440-5691-451a-b402-8eb5af4a9793","Type":"ContainerStarted","Data":"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"}
Jul 27 21:42:01.865207 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:01.864639    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" event={"ID":"29e52440-5691-451a-b402-8eb5af4a9793","Type":"ContainerStarted","Data":"78ffc389df11230bb61fb348ce36311a1c4eabe2e564d07af22f9e7387ef6808"}
Jul 27 21:42:01.865207 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:01.864670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:01.884457 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:01.884401    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" podStartSLOduration=2.469424336 podStartE2EDuration="2.884384674s" totalImagesPullingTime="414.960338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:42:01.882191682 +0000 UTC m=+3027.656124106" watchObservedRunningTime="2026-07-27 21:42:01.884384674 +0000 UTC m=+3027.658317095"
Jul 27 21:42:04.426960 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:04.426919    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:07.875665 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:07.875630    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:25.978918 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:25.978824    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:42:25.979388 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:25.979099    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" podUID="29e52440-5691-451a-b402-8eb5af4a9793" containerName="maas-api" containerID="cri-o://e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1" gracePeriod=30
Jul 27 21:42:26.015391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.015357    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:26.099507 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.099476    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:26.181726 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.181692    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:26.238106 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.238036    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:26.316926 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.316887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv2t6\" (UniqueName: \"kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6\") pod \"29e52440-5691-451a-b402-8eb5af4a9793\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") "
Jul 27 21:42:26.316926 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.316926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca\") pod \"29e52440-5691-451a-b402-8eb5af4a9793\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") "
Jul 27 21:42:26.317173 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.316944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") pod \"29e52440-5691-451a-b402-8eb5af4a9793\" (UID: \"29e52440-5691-451a-b402-8eb5af4a9793\") "
Jul 27 21:42:26.317322 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.317298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29e52440-5691-451a-b402-8eb5af4a9793" (UID: "29e52440-5691-451a-b402-8eb5af4a9793"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:26.319267 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.319247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6" (OuterVolumeSpecName: "kube-api-access-mv2t6") pod "29e52440-5691-451a-b402-8eb5af4a9793" (UID: "29e52440-5691-451a-b402-8eb5af4a9793"). InnerVolumeSpecName "kube-api-access-mv2t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:26.319420 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.319398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29e52440-5691-451a-b402-8eb5af4a9793" (UID: "29e52440-5691-451a-b402-8eb5af4a9793"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:42:26.417596 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.417565    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv2t6\" (UniqueName: \"kubernetes.io/projected/29e52440-5691-451a-b402-8eb5af4a9793-kube-api-access-mv2t6\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:42:26.417596 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.417592    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e52440-5691-451a-b402-8eb5af4a9793-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:42:26.417596 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:26.417603    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e52440-5691-451a-b402-8eb5af4a9793-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:42:27.001999 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.001965    2580 generic.go:356] "Generic (PLEG): container finished" podID="29e52440-5691-451a-b402-8eb5af4a9793" containerID="e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1" exitCode=0
Jul 27 21:42:27.002565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.002013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"
Jul 27 21:42:27.002565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.002057    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" event={"ID":"29e52440-5691-451a-b402-8eb5af4a9793","Type":"ContainerDied","Data":"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"}
Jul 27 21:42:27.002565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.002140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp" event={"ID":"29e52440-5691-451a-b402-8eb5af4a9793","Type":"ContainerDied","Data":"78ffc389df11230bb61fb348ce36311a1c4eabe2e564d07af22f9e7387ef6808"}
Jul 27 21:42:27.002565 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.002176    2580 scope.go:117] "RemoveContainer" containerID="e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"
Jul 27 21:42:27.011482 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.011456    2580 scope.go:117] "RemoveContainer" containerID="e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"
Jul 27 21:42:27.011785 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:42:27.011769    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1\": container with ID starting with e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1 not found: ID does not exist" containerID="e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"
Jul 27 21:42:27.011842 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.011791    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1"} err="failed to get container status \"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1\": rpc error: code = NotFound desc = could not find container \"e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1\": container with ID starting with e75933675da5622f9676b1b536cc6c44ebbe06ebf645a14abfb0228d4e0ebdf1 not found: ID does not exist"
Jul 27 21:42:27.019589 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.019556    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:42:27.023893 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:27.023866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-400a5b-6b467ddb66-z72sp"]
Jul 27 21:42:28.805725 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:28.805686    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e52440-5691-451a-b402-8eb5af4a9793" path="/var/lib/kubelet/pods/29e52440-5691-451a-b402-8eb5af4a9793/volumes"
Jul 27 21:42:30.531517 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:30.531473    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:30.954721 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:30.954634    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:37.203442 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:37.203401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:37.306121 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:37.306083    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:37.541183 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:37.541142    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:37.788299 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:37.788261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:57.136724 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.136684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:57.221719 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.221682    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:42:57.221947 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.221923    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" podUID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" containerName="payload-pre-processing" containerID="cri-o://14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8" gracePeriod=30
Jul 27 21:42:57.297620 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.297589    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:57.458113 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.458067    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:57.486032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.486010    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:42:57.611357 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.611331    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxd88\" (UniqueName: \"kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88\") pod \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") "
Jul 27 21:42:57.611533 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.611431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume\") pod \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\" (UID: \"18f14f63-b7cb-4971-9fc9-f659d3de9d68\") "
Jul 27 21:42:57.611836 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.611803    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18f14f63-b7cb-4971-9fc9-f659d3de9d68" (UID: "18f14f63-b7cb-4971-9fc9-f659d3de9d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:57.613659 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.613638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88" (OuterVolumeSpecName: "kube-api-access-hxd88") pod "18f14f63-b7cb-4971-9fc9-f659d3de9d68" (UID: "18f14f63-b7cb-4971-9fc9-f659d3de9d68"). InnerVolumeSpecName "kube-api-access-hxd88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:57.712566 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.712536    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxd88\" (UniqueName: \"kubernetes.io/projected/18f14f63-b7cb-4971-9fc9-f659d3de9d68-kube-api-access-hxd88\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:42:57.712566 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:57.712563    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f14f63-b7cb-4971-9fc9-f659d3de9d68-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:42:58.166264 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.166153    2580 generic.go:356] "Generic (PLEG): container finished" podID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" containerID="14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8" exitCode=0
Jul 27 21:42:58.166264 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.166195    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"
Jul 27 21:42:58.166264 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.166246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" event={"ID":"18f14f63-b7cb-4971-9fc9-f659d3de9d68","Type":"ContainerDied","Data":"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"}
Jul 27 21:42:58.166817 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.166310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc" event={"ID":"18f14f63-b7cb-4971-9fc9-f659d3de9d68","Type":"ContainerDied","Data":"abe455e2f8fc250e95f531afb3b8ad866346bff571ba012fb405ec15236bb728"}
Jul 27 21:42:58.166817 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.166342    2580 scope.go:117] "RemoveContainer" containerID="14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"
Jul 27 21:42:58.176032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.176012    2580 scope.go:117] "RemoveContainer" containerID="14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"
Jul 27 21:42:58.176307 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:42:58.176288    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8\": container with ID starting with 14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8 not found: ID does not exist" containerID="14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"
Jul 27 21:42:58.176369 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.176318    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8"} err="failed to get container status \"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8\": rpc error: code = NotFound desc = could not find container \"14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8\": container with ID starting with 14a92546234135316532df98f2843b2fb9ca24713c9028d216484b170ace95a8 not found: ID does not exist"
Jul 27 21:42:58.188102 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.188083    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:42:58.194003 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.193979    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b4bf5c-65b6ffc45b-qgkzc"]
Jul 27 21:42:58.805058 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:42:58.805022    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" path="/var/lib/kubelet/pods/18f14f63-b7cb-4971-9fc9-f659d3de9d68/volumes"
Jul 27 21:43:01.747011 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:01.746976    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:02.166079 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:02.165985    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:07.154632 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:07.154590    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:43:07.251397 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:07.251359    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:18.890902 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:18.890866    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:18.998356 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:18.998321    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:22.174953 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.174900    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" podUID="64965432-3f06-4c60-a612-cd08f5016649" containerName="main" containerID="cri-o://ef3cce4861d0d19d3379db57532606128f1b3e3f722d2d55c1a09260eb052e15" gracePeriod=45
Jul 27 21:43:22.303270 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.303235    2580 generic.go:356] "Generic (PLEG): container finished" podID="64965432-3f06-4c60-a612-cd08f5016649" containerID="ef3cce4861d0d19d3379db57532606128f1b3e3f722d2d55c1a09260eb052e15" exitCode=0
Jul 27 21:43:22.303461 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.303303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" event={"ID":"64965432-3f06-4c60-a612-cd08f5016649","Type":"ContainerDied","Data":"ef3cce4861d0d19d3379db57532606128f1b3e3f722d2d55c1a09260eb052e15"}
Jul 27 21:43:22.441548 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.441478    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:43:22.571735 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571695    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.571943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.571943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571803    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.571943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571861    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.571943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.571943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.571922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdd8h\" (UniqueName: \"kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h\") pod \"64965432-3f06-4c60-a612-cd08f5016649\" (UID: \"64965432-3f06-4c60-a612-cd08f5016649\") "
Jul 27 21:43:22.572302 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:22.572302 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache" (OuterVolumeSpecName: "model-cache") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:22.572302 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home" (OuterVolumeSpecName: "home") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:22.572436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572329    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-home\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:22.572436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572350    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-model-cache\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:22.572436 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.572366    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-tmp-dir\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:22.574307 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.574272    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h" (OuterVolumeSpecName: "kube-api-access-xdd8h") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "kube-api-access-xdd8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:22.574307 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.574286    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm" (OuterVolumeSpecName: "dshm") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:22.574438 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.574333    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64965432-3f06-4c60-a612-cd08f5016649" (UID: "64965432-3f06-4c60-a612-cd08f5016649"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:43:22.673465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.673422    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdd8h\" (UniqueName: \"kubernetes.io/projected/64965432-3f06-4c60-a612-cd08f5016649-kube-api-access-xdd8h\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:22.673465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.673456    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64965432-3f06-4c60-a612-cd08f5016649-tls-certs\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:22.673465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:22.673467    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64965432-3f06-4c60-a612-cd08f5016649-dshm\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:43:23.309977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:23.309947    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"
Jul 27 21:43:23.309977 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:23.309965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d" event={"ID":"64965432-3f06-4c60-a612-cd08f5016649","Type":"ContainerDied","Data":"46a90b20e1ca59ea11b003a2e8d3f093058286b132429cedff1893a55a60e4f9"}
Jul 27 21:43:23.310486 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:23.310010    2580 scope.go:117] "RemoveContainer" containerID="ef3cce4861d0d19d3379db57532606128f1b3e3f722d2d55c1a09260eb052e15"
Jul 27 21:43:23.326866 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:23.326839    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:43:23.330618 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:23.330589    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b4bf5c/ipp-route-b4bf5c-kserve-6455c974b7-bx25d"]
Jul 27 21:43:24.805763 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:24.805720    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64965432-3f06-4c60-a612-cd08f5016649" path="/var/lib/kubelet/pods/64965432-3f06-4c60-a612-cd08f5016649/volumes"
Jul 27 21:43:25.858855 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:25.858819    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:43.660521 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:43.660485    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:44.262960 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:44.262914    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:49.791341 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:49.791301    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:49.895545 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:49.895508    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:50.101828 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:50.101742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:50.366493 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:43:50.366405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:01.043534 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.043454    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:44:01.043922 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.043715    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" podUID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" containerName="maas-api" containerID="cri-o://1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1" gracePeriod=30
Jul 27 21:44:01.113087 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.113049    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:01.184002 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.183963    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:44:01.184296 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.184249    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" podUID="7c421612-18c9-452b-b911-46e6a124bed1" containerName="payload-processing" containerID="cri-o://0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407" gracePeriod=30
Jul 27 21:44:01.222443 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.222405    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:01.319637 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.319505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:01.328691 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.328662    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:44:01.434744 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.434699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mclb8\" (UniqueName: \"kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8\") pod \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") "
Jul 27 21:44:01.434939 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.434757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca\") pod \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") "
Jul 27 21:44:01.434939 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.434814    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls\") pod \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\" (UID: \"cb52d9c9-27f4-42b7-8254-cd2cecf81984\") "
Jul 27 21:44:01.435152 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.435124    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb52d9c9-27f4-42b7-8254-cd2cecf81984" (UID: "cb52d9c9-27f4-42b7-8254-cd2cecf81984"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:01.437196 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.437160    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb52d9c9-27f4-42b7-8254-cd2cecf81984" (UID: "cb52d9c9-27f4-42b7-8254-cd2cecf81984"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:01.437440 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.437280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8" (OuterVolumeSpecName: "kube-api-access-mclb8") pod "cb52d9c9-27f4-42b7-8254-cd2cecf81984" (UID: "cb52d9c9-27f4-42b7-8254-cd2cecf81984"). InnerVolumeSpecName "kube-api-access-mclb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:01.456803 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.456769    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:44:01.507986 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.507941    2580 generic.go:356] "Generic (PLEG): container finished" podID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" containerID="1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1" exitCode=0
Jul 27 21:44:01.507986 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.507979    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"
Jul 27 21:44:01.508290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.508020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" event={"ID":"cb52d9c9-27f4-42b7-8254-cd2cecf81984","Type":"ContainerDied","Data":"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"}
Jul 27 21:44:01.508290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.508086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6" event={"ID":"cb52d9c9-27f4-42b7-8254-cd2cecf81984","Type":"ContainerDied","Data":"8039e1301989700527e1a08e7ac47039ac307127dfafd4084f04c5a3a9784107"}
Jul 27 21:44:01.508290 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.508120    2580 scope.go:117] "RemoveContainer" containerID="1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"
Jul 27 21:44:01.509503 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.509448    2580 generic.go:356] "Generic (PLEG): container finished" podID="7c421612-18c9-452b-b911-46e6a124bed1" containerID="0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407" exitCode=0
Jul 27 21:44:01.509503 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.509478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" event={"ID":"7c421612-18c9-452b-b911-46e6a124bed1","Type":"ContainerDied","Data":"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"}
Jul 27 21:44:01.509645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.509520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq" event={"ID":"7c421612-18c9-452b-b911-46e6a124bed1","Type":"ContainerDied","Data":"7d35e9c51b45c0426c1965fa02d293f4553aaf6a45033320e6646475e54eab7c"}
Jul 27 21:44:01.509645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.509543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"
Jul 27 21:44:01.519049 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.519028    2580 scope.go:117] "RemoveContainer" containerID="1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"
Jul 27 21:44:01.519366 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:01.519349    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1\": container with ID starting with 1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1 not found: ID does not exist" containerID="1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"
Jul 27 21:44:01.519429 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.519376    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1"} err="failed to get container status \"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1\": rpc error: code = NotFound desc = could not find container \"1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1\": container with ID starting with 1015ab075e85789956914a9332440a9c06fc598599b337fcc6d9cfac8486c9f1 not found: ID does not exist"
Jul 27 21:44:01.519429 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.519393    2580 scope.go:117] "RemoveContainer" containerID="0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"
Jul 27 21:44:01.528028 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.528008    2580 scope.go:117] "RemoveContainer" containerID="0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"
Jul 27 21:44:01.528499 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:01.528456    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407\": container with ID starting with 0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407 not found: ID does not exist" containerID="0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"
Jul 27 21:44:01.528602 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.528507    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407"} err="failed to get container status \"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407\": rpc error: code = NotFound desc = could not find container \"0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407\": container with ID starting with 0153dee6680e8a69e1284b31299b63730d28069f291a461751ff8805cfc42407 not found: ID does not exist"
Jul 27 21:44:01.531006 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.530981    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:44:01.535258 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535234    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9e51a0-574cd744bf-lrkr6"]
Jul 27 21:44:01.535484 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535466    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume\") pod \"7c421612-18c9-452b-b911-46e6a124bed1\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") "
Jul 27 21:44:01.535537 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp8jz\" (UniqueName: \"kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz\") pod \"7c421612-18c9-452b-b911-46e6a124bed1\" (UID: \"7c421612-18c9-452b-b911-46e6a124bed1\") "
Jul 27 21:44:01.535892 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535876    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclb8\" (UniqueName: \"kubernetes.io/projected/cb52d9c9-27f4-42b7-8254-cd2cecf81984-kube-api-access-mclb8\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:01.535937 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535903    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb52d9c9-27f4-42b7-8254-cd2cecf81984-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:01.535937 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535915    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb52d9c9-27f4-42b7-8254-cd2cecf81984-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:01.535937 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.535880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c421612-18c9-452b-b911-46e6a124bed1" (UID: "7c421612-18c9-452b-b911-46e6a124bed1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:01.537744 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.537728    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz" (OuterVolumeSpecName: "kube-api-access-sp8jz") pod "7c421612-18c9-452b-b911-46e6a124bed1" (UID: "7c421612-18c9-452b-b911-46e6a124bed1"). InnerVolumeSpecName "kube-api-access-sp8jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:01.636766 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.636672    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c421612-18c9-452b-b911-46e6a124bed1-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:01.636766 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.636703    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8jz\" (UniqueName: \"kubernetes.io/projected/7c421612-18c9-452b-b911-46e6a124bed1-kube-api-access-sp8jz\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:01.832487 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.832447    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:44:01.838798 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:01.838765    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e51a0-5b98c6b49d-qc8tq"]
Jul 27 21:44:02.804936 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:02.804897    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c421612-18c9-452b-b911-46e6a124bed1" path="/var/lib/kubelet/pods/7c421612-18c9-452b-b911-46e6a124bed1/volumes"
Jul 27 21:44:02.805354 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:02.805327    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" path="/var/lib/kubelet/pods/cb52d9c9-27f4-42b7-8254-cd2cecf81984/volumes"
Jul 27 21:44:05.581756 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:05.581716    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:05.971316 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:05.971286    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:11.870794 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:11.870748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:11.971004 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:11.970974    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:12.155473 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:12.155379    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:12.458119 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:12.458086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:22.997470 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:22.997424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:44:22.999780 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:22.997680    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" podUID="c4719105-3444-4894-be97-1e3f133309f4" containerName="maas-api" containerID="cri-o://02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483" gracePeriod=30
Jul 27 21:44:23.024526 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.024494    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:23.056872 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.056833    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:44:23.057505 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.057456    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" podUID="4c2f38d2-394d-4e24-8968-888489dcabe8" containerName="payload-pre-processing" containerID="cri-o://80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8" gracePeriod=30
Jul 27 21:44:23.126517 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.126475    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:23.226604 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.226182    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:23.310724 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.310698    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:44:23.346728 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.346704    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:44:23.456141 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls\") pod \"c4719105-3444-4894-be97-1e3f133309f4\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") "
Jul 27 21:44:23.456141 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca\") pod \"c4719105-3444-4894-be97-1e3f133309f4\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") "
Jul 27 21:44:23.456391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456170    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89kvk\" (UniqueName: \"kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk\") pod \"4c2f38d2-394d-4e24-8968-888489dcabe8\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") "
Jul 27 21:44:23.456391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume\") pod \"4c2f38d2-394d-4e24-8968-888489dcabe8\" (UID: \"4c2f38d2-394d-4e24-8968-888489dcabe8\") "
Jul 27 21:44:23.456391 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg9kj\" (UniqueName: \"kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj\") pod \"c4719105-3444-4894-be97-1e3f133309f4\" (UID: \"c4719105-3444-4894-be97-1e3f133309f4\") "
Jul 27 21:44:23.456609 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4719105-3444-4894-be97-1e3f133309f4" (UID: "c4719105-3444-4894-be97-1e3f133309f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:23.456689 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.456646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c2f38d2-394d-4e24-8968-888489dcabe8" (UID: "4c2f38d2-394d-4e24-8968-888489dcabe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:23.458597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.458568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk" (OuterVolumeSpecName: "kube-api-access-89kvk") pod "4c2f38d2-394d-4e24-8968-888489dcabe8" (UID: "4c2f38d2-394d-4e24-8968-888489dcabe8"). InnerVolumeSpecName "kube-api-access-89kvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:23.458597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.458575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4719105-3444-4894-be97-1e3f133309f4" (UID: "c4719105-3444-4894-be97-1e3f133309f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:23.458597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.458584    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj" (OuterVolumeSpecName: "kube-api-access-qg9kj") pod "c4719105-3444-4894-be97-1e3f133309f4" (UID: "c4719105-3444-4894-be97-1e3f133309f4"). InnerVolumeSpecName "kube-api-access-qg9kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:23.557629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.557510    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4719105-3444-4894-be97-1e3f133309f4-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:23.557629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.557558    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4719105-3444-4894-be97-1e3f133309f4-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:23.557629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.557573    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89kvk\" (UniqueName: \"kubernetes.io/projected/4c2f38d2-394d-4e24-8968-888489dcabe8-kube-api-access-89kvk\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:23.557629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.557587    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2f38d2-394d-4e24-8968-888489dcabe8-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:23.557629 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.557599    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg9kj\" (UniqueName: \"kubernetes.io/projected/c4719105-3444-4894-be97-1e3f133309f4-kube-api-access-qg9kj\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:23.621489 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.621445    2580 generic.go:356] "Generic (PLEG): container finished" podID="c4719105-3444-4894-be97-1e3f133309f4" containerID="02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483" exitCode=0
Jul 27 21:44:23.621489 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.621493    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"
Jul 27 21:44:23.621747 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.621534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" event={"ID":"c4719105-3444-4894-be97-1e3f133309f4","Type":"ContainerDied","Data":"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"}
Jul 27 21:44:23.621747 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.621581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh" event={"ID":"c4719105-3444-4894-be97-1e3f133309f4","Type":"ContainerDied","Data":"d8be0b7851869e63e0363aee48c312a5742f3122d3a23af2de2e7974e7211698"}
Jul 27 21:44:23.621747 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.621610    2580 scope.go:117] "RemoveContainer" containerID="02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"
Jul 27 21:44:23.622994 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.622974    2580 generic.go:356] "Generic (PLEG): container finished" podID="4c2f38d2-394d-4e24-8968-888489dcabe8" containerID="80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8" exitCode=0
Jul 27 21:44:23.623080 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.623058    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"
Jul 27 21:44:23.623147 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.623058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" event={"ID":"4c2f38d2-394d-4e24-8968-888489dcabe8","Type":"ContainerDied","Data":"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"}
Jul 27 21:44:23.623193 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.623182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f" event={"ID":"4c2f38d2-394d-4e24-8968-888489dcabe8","Type":"ContainerDied","Data":"102215759eeb13b5c6b6aa75c5687bfd166e388a578d549b772dc65d02aa2f91"}
Jul 27 21:44:23.633037 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.633017    2580 scope.go:117] "RemoveContainer" containerID="02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"
Jul 27 21:44:23.633382 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:23.633363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483\": container with ID starting with 02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483 not found: ID does not exist" containerID="02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"
Jul 27 21:44:23.633495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.633391    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483"} err="failed to get container status \"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483\": rpc error: code = NotFound desc = could not find container \"02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483\": container with ID starting with 02aa31685e25035230ba1a40fd0f8bbcd17c69a6e7a278b0359189306c5bd483 not found: ID does not exist"
Jul 27 21:44:23.633495 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.633416    2580 scope.go:117] "RemoveContainer" containerID="80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"
Jul 27 21:44:23.642945 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.642922    2580 scope.go:117] "RemoveContainer" containerID="80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"
Jul 27 21:44:23.643263 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:23.643234    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8\": container with ID starting with 80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8 not found: ID does not exist" containerID="80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"
Jul 27 21:44:23.643328 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.643275    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8"} err="failed to get container status \"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8\": rpc error: code = NotFound desc = could not find container \"80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8\": container with ID starting with 80a05735411a86df6edbe9f0790472f4d11f3a410ebbfe10206d7de88021f7a8 not found: ID does not exist"
Jul 27 21:44:23.644966 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.644947    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:44:23.653180 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.653152    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-6dd721-74679586b9-cctnh"]
Jul 27 21:44:23.665007 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.664978    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:44:23.671123 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:23.671095    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-6dd721-546748874d-8lx7f"]
Jul 27 21:44:24.805525 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:24.805482    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2f38d2-394d-4e24-8968-888489dcabe8" path="/var/lib/kubelet/pods/4c2f38d2-394d-4e24-8968-888489dcabe8/volumes"
Jul 27 21:44:24.805953 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:24.805938    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4719105-3444-4894-be97-1e3f133309f4" path="/var/lib/kubelet/pods/c4719105-3444-4894-be97-1e3f133309f4/volumes"
Jul 27 21:44:27.641175 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:27.641147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:28.024903 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:28.024864    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:33.900182 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:33.900146    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:33.962349 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:33.962316    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:34.184424 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:34.184340    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:34.441627 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:34.441538    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:44.925044 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:44.925011    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:44:44.925509 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:44.925241    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" podUID="590072cb-2465-48d2-a59e-324390e51fd0" containerName="maas-api" containerID="cri-o://a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9" gracePeriod=30
Jul 27 21:44:44.955696 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:44.955666    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:44.980759 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:44.980729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:44:44.980972 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:44.980949    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" podUID="6f220613-d4de-4196-8bc3-6a91db4f7b98" containerName="payload-processing" containerID="cri-o://c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0" gracePeriod=30
Jul 27 21:44:45.055951 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.055920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:45.112645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.112613    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:45.258154 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.258128    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:44:45.273506 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.273455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:44:45.353943 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.353908    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls\") pod \"590072cb-2465-48d2-a59e-324390e51fd0\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") "
Jul 27 21:44:45.354146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.353964    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca\") pod \"590072cb-2465-48d2-a59e-324390e51fd0\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") "
Jul 27 21:44:45.354146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.353996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume\") pod \"6f220613-d4de-4196-8bc3-6a91db4f7b98\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") "
Jul 27 21:44:45.354146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.354040    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvjzh\" (UniqueName: \"kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh\") pod \"590072cb-2465-48d2-a59e-324390e51fd0\" (UID: \"590072cb-2465-48d2-a59e-324390e51fd0\") "
Jul 27 21:44:45.354146 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.354094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtl7b\" (UniqueName: \"kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b\") pod \"6f220613-d4de-4196-8bc3-6a91db4f7b98\" (UID: \"6f220613-d4de-4196-8bc3-6a91db4f7b98\") "
Jul 27 21:44:45.354415 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.354388    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "590072cb-2465-48d2-a59e-324390e51fd0" (UID: "590072cb-2465-48d2-a59e-324390e51fd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:45.354465 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.354422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f220613-d4de-4196-8bc3-6a91db4f7b98" (UID: "6f220613-d4de-4196-8bc3-6a91db4f7b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:45.356395 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.356368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "590072cb-2465-48d2-a59e-324390e51fd0" (UID: "590072cb-2465-48d2-a59e-324390e51fd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:45.356395 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.356392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh" (OuterVolumeSpecName: "kube-api-access-gvjzh") pod "590072cb-2465-48d2-a59e-324390e51fd0" (UID: "590072cb-2465-48d2-a59e-324390e51fd0"). InnerVolumeSpecName "kube-api-access-gvjzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:45.356536 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.356416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b" (OuterVolumeSpecName: "kube-api-access-jtl7b") pod "6f220613-d4de-4196-8bc3-6a91db4f7b98" (UID: "6f220613-d4de-4196-8bc3-6a91db4f7b98"). InnerVolumeSpecName "kube-api-access-jtl7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:45.455032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.454944    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtl7b\" (UniqueName: \"kubernetes.io/projected/6f220613-d4de-4196-8bc3-6a91db4f7b98-kube-api-access-jtl7b\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:45.455032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.454976    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/590072cb-2465-48d2-a59e-324390e51fd0-maas-api-tls\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:45.455032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.454986    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/590072cb-2465-48d2-a59e-324390e51fd0-openshift-service-ca\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:45.455032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.454997    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f220613-d4de-4196-8bc3-6a91db4f7b98-plugins-config-volume\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:45.455032 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.455006    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvjzh\" (UniqueName: \"kubernetes.io/projected/590072cb-2465-48d2-a59e-324390e51fd0-kube-api-access-gvjzh\") on node \"ip-10-0-131-89.ec2.internal\" DevicePath \"\""
Jul 27 21:44:45.739370 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.739335    2580 generic.go:356] "Generic (PLEG): container finished" podID="590072cb-2465-48d2-a59e-324390e51fd0" containerID="a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9" exitCode=0
Jul 27 21:44:45.739557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.739394    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"
Jul 27 21:44:45.739557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.739422    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" event={"ID":"590072cb-2465-48d2-a59e-324390e51fd0","Type":"ContainerDied","Data":"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"}
Jul 27 21:44:45.739557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.739504    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj" event={"ID":"590072cb-2465-48d2-a59e-324390e51fd0","Type":"ContainerDied","Data":"743d8b1fb6c682031fea5a9e488b66d69bb9e1f49d0862fe120d4427a346d0d6"}
Jul 27 21:44:45.739557 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.739535    2580 scope.go:117] "RemoveContainer" containerID="a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"
Jul 27 21:44:45.740959 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.740919    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f220613-d4de-4196-8bc3-6a91db4f7b98" containerID="c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0" exitCode=0
Jul 27 21:44:45.741065 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.740950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" event={"ID":"6f220613-d4de-4196-8bc3-6a91db4f7b98","Type":"ContainerDied","Data":"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"}
Jul 27 21:44:45.741065 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.740978    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"
Jul 27 21:44:45.741065 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.741019    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8" event={"ID":"6f220613-d4de-4196-8bc3-6a91db4f7b98","Type":"ContainerDied","Data":"0b384484ef5beb59c69fe0de8fdf6fba3b7691dd7138565b9eac1860e9a0a86d"}
Jul 27 21:44:45.750272 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.750250    2580 scope.go:117] "RemoveContainer" containerID="a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"
Jul 27 21:44:45.750569 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:45.750548    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9\": container with ID starting with a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9 not found: ID does not exist" containerID="a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"
Jul 27 21:44:45.750650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.750579    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9"} err="failed to get container status \"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9\": rpc error: code = NotFound desc = could not find container \"a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9\": container with ID starting with a13e863624aef5005933e598a4990cee0bdf21089053b9238e79a989eacaecd9 not found: ID does not exist"
Jul 27 21:44:45.750650 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.750601    2580 scope.go:117] "RemoveContainer" containerID="c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"
Jul 27 21:44:45.758802 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.758783    2580 scope.go:117] "RemoveContainer" containerID="c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"
Jul 27 21:44:45.759060 ip-10-0-131-89 kubenswrapper[2580]: E0727 21:44:45.759039    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0\": container with ID starting with c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0 not found: ID does not exist" containerID="c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"
Jul 27 21:44:45.759116 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.759067    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0"} err="failed to get container status \"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0\": rpc error: code = NotFound desc = could not find container \"c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0\": container with ID starting with c1fc3d5c5d9b2b2216e69d93b9083159da4b81aa8cd51510aabdca397a4ab3e0 not found: ID does not exist"
Jul 27 21:44:45.763368 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.763348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:44:45.766939 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.766918    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5f50cb-84b77964c-lf9dj"]
Jul 27 21:44:45.779635 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.779610    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:44:45.785227 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:45.785183    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5f50cb-5b647c89d-tvzb8"]
Jul 27 21:44:46.804831 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:46.804796    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590072cb-2465-48d2-a59e-324390e51fd0" path="/var/lib/kubelet/pods/590072cb-2465-48d2-a59e-324390e51fd0/volumes"
Jul 27 21:44:46.805190 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:46.805149    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f220613-d4de-4196-8bc3-6a91db4f7b98" path="/var/lib/kubelet/pods/6f220613-d4de-4196-8bc3-6a91db4f7b98/volumes"
Jul 27 21:44:49.630916 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:49.630879    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:50.024781 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:50.024748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:55.944741 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:55.944711    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:56.045078 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:56.045042    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:56.226053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:56.226013    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:56.287833 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:56.287801    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:56.474553 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:44:56.474520    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:03.123398 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:03.123360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55f67d7fff-6p9ql_783e4803-42ff-47e9-b2c8-25dafefec617/manager/0.log"
Jul 27 21:45:03.683890 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:03.683858    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-dqcxf_2d46802e-6d8d-41b1-86af-811c999d9b9e/copy-manifests/0.log"
Jul 27 21:45:03.700331 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:03.700303    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-dqcxf_2d46802e-6d8d-41b1-86af-811c999d9b9e/manager/0.log"
Jul 27 21:45:04.209814 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:04.209780    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5b64468d9c-hqrbc_357eb02d-52e6-4fe3-874d-d5df4d85b887/maas-api/0.log"
Jul 27 21:45:04.716701 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:04.716672    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kblm2_15c864a5-74fc-4826-9dc6-1b704acb6e63/postgres/0.log"
Jul 27 21:45:07.661838 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:07.661805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-n955m_03ae1502-2e08-4c91-8ce1-4db469bdbaed/kube-auth-proxy/0.log"
Jul 27 21:45:08.034986 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:08.034951    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pxn4h_a9ffe201-d02b-4136-893d-656f80bb5e36/payload-pre-processing/0.log"
Jul 27 21:45:08.161668 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:08.161634    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:45:08.303625 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:08.303547    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6b4bf9c-jn462_72c0faa1-cedb-4a19-838f-6e37a26255e3/router/0.log"
Jul 27 21:45:08.933255 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:08.933226    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ql9qx_83986328-9388-4556-b86f-b971c9d17d98/main/0.log"
Jul 27 21:45:09.183051 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:09.183025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5cskg_dd1b7350-a42d-4da0-ba85-68a9b84443fc/main/0.log"
Jul 27 21:45:09.311645 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:09.311618    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-vv9rj_22a2be61-e35c-425f-825d-a4462ea0a390/main/0.log"
Jul 27 21:45:15.425600 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:15.425560    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:16.839697 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:16.839662    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:18.319982 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:18.319947    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m582w_bd91b7f1-6d7e-4cfd-a400-373508ac1c0c/global-pull-secret-syncer/0.log"
Jul 27 21:45:18.401639 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:18.401604    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b82kq_d3d573c1-2c7d-493e-bb05-f5c607222edd/konnectivity-agent/0.log"
Jul 27 21:45:18.501840 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:18.501810    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-89.ec2.internal_9cbe27aeb4e99555881d264eb730f693/haproxy/0.log"
Jul 27 21:45:26.390502 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.390469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-prdpn_251ecbce-92e2-486d-8292-8f593eeac49a/monitoring-plugin/0.log"
Jul 27 21:45:26.423065 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.423035    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pzqf_f0fc842b-d61e-47d4-9630-b9cd0ba04117/node-exporter/0.log"
Jul 27 21:45:26.448311 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.448287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pzqf_f0fc842b-d61e-47d4-9630-b9cd0ba04117/kube-rbac-proxy/0.log"
Jul 27 21:45:26.472085 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.472066    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pzqf_f0fc842b-d61e-47d4-9630-b9cd0ba04117/init-textfile/0.log"
Jul 27 21:45:26.649978 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.649906    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m69rm_63a23771-4518-48de-95d5-92b93d00d0a3/kube-rbac-proxy-main/0.log"
Jul 27 21:45:26.672246 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.672206    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m69rm_63a23771-4518-48de-95d5-92b93d00d0a3/kube-rbac-proxy-self/0.log"
Jul 27 21:45:26.695599 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.695576    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-m69rm_63a23771-4518-48de-95d5-92b93d00d0a3/openshift-state-metrics/0.log"
Jul 27 21:45:26.741621 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.741595    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/prometheus/0.log"
Jul 27 21:45:26.769520 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.769491    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/config-reloader/0.log"
Jul 27 21:45:26.797654 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.797626    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/thanos-sidecar/0.log"
Jul 27 21:45:26.822628 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.822600    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/kube-rbac-proxy-web/0.log"
Jul 27 21:45:26.847260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.847235    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/kube-rbac-proxy/0.log"
Jul 27 21:45:26.870896 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.870863    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/kube-rbac-proxy-thanos/0.log"
Jul 27 21:45:26.892949 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.892922    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb56d0eb-a001-4385-ab59-ea3d04b27837/init-config-reloader/0.log"
Jul 27 21:45:26.922516 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.922429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kjhrt_d2ea68fd-2947-4918-bb1a-7b4b4669b768/prometheus-operator/0.log"
Jul 27 21:45:26.944167 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.944137    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kjhrt_d2ea68fd-2947-4918-bb1a-7b4b4669b768/kube-rbac-proxy/0.log"
Jul 27 21:45:26.969052 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:26.969019    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-q2qcp_39b51e8b-e3ea-46c8-a564-e8435cee15b4/prometheus-operator-admission-webhook/0.log"
Jul 27 21:45:27.970605 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970574    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"]
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970947    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2f38d2-394d-4e24-8968-888489dcabe8" containerName="payload-pre-processing"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970959    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2f38d2-394d-4e24-8968-888489dcabe8" containerName="payload-pre-processing"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970968    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" containerName="payload-pre-processing"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970976    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" containerName="payload-pre-processing"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970987    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.970996    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971005    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590072cb-2465-48d2-a59e-324390e51fd0" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971010    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="590072cb-2465-48d2-a59e-324390e51fd0" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971018    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e52440-5691-451a-b402-8eb5af4a9793" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971023    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e52440-5691-451a-b402-8eb5af4a9793" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971030    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4719105-3444-4894-be97-1e3f133309f4" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971035    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4719105-3444-4894-be97-1e3f133309f4" containerName="maas-api"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971047    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64965432-3f06-4c60-a612-cd08f5016649" containerName="main"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="64965432-3f06-4c60-a612-cd08f5016649" containerName="main"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971061    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f220613-d4de-4196-8bc3-6a91db4f7b98" containerName="payload-processing"
Jul 27 21:45:27.971064 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971067    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f220613-d4de-4196-8bc3-6a91db4f7b98" containerName="payload-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971074    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c421612-18c9-452b-b911-46e6a124bed1" containerName="payload-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971079    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c421612-18c9-452b-b911-46e6a124bed1" containerName="payload-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971133    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2f38d2-394d-4e24-8968-888489dcabe8" containerName="payload-pre-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971141    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="590072cb-2465-48d2-a59e-324390e51fd0" containerName="maas-api"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971148    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f220613-d4de-4196-8bc3-6a91db4f7b98" containerName="payload-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971153    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c421612-18c9-452b-b911-46e6a124bed1" containerName="payload-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971162    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e52440-5691-451a-b402-8eb5af4a9793" containerName="maas-api"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971168    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4719105-3444-4894-be97-1e3f133309f4" containerName="maas-api"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971173    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18f14f63-b7cb-4971-9fc9-f659d3de9d68" containerName="payload-pre-processing"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971178    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="64965432-3f06-4c60-a612-cd08f5016649" containerName="main"
Jul 27 21:45:27.971575 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.971185    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb52d9c9-27f4-42b7-8254-cd2cecf81984" containerName="maas-api"
Jul 27 21:45:27.974384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.974363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:27.976887 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.976858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd6n7\"/\"kube-root-ca.crt\""
Jul 27 21:45:27.976998 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.976908    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd6n7\"/\"openshift-service-ca.crt\""
Jul 27 21:45:27.977705 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.977689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wd6n7\"/\"default-dockercfg-pmb4h\""
Jul 27 21:45:27.983354 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:27.983329    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"]
Jul 27 21:45:28.042616 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.042578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrtz\" (UniqueName: \"kubernetes.io/projected/fa074b02-4fda-479f-80b2-023cae3f7db8-kube-api-access-htrtz\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.042825 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.042697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-proc\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.042825 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.042744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-sys\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.042825 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.042785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-lib-modules\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.042953 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.042844    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-podres\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.143797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-proc\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.143797 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-sys\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-lib-modules\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-podres\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrtz\" (UniqueName: \"kubernetes.io/projected/fa074b02-4fda-479f-80b2-023cae3f7db8-kube-api-access-htrtz\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-proc\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143902    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-sys\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-lib-modules\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.144053 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.143987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa074b02-4fda-479f-80b2-023cae3f7db8-podres\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.152800 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.152768    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrtz\" (UniqueName: \"kubernetes.io/projected/fa074b02-4fda-479f-80b2-023cae3f7db8-kube-api-access-htrtz\") pod \"perf-node-gather-daemonset-xtlmm\" (UID: \"fa074b02-4fda-479f-80b2-023cae3f7db8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.285439 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.285356    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.422430 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.422406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"]
Jul 27 21:45:28.424892 ip-10-0-131-89 kubenswrapper[2580]: W0727 21:45:28.424857    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa074b02_4fda_479f_80b2_023cae3f7db8.slice/crio-dd5854a7dfbd63ad22102624c3a429062c53a69df4471e882d41b55fea2962bf WatchSource:0}: Error finding container dd5854a7dfbd63ad22102624c3a429062c53a69df4471e882d41b55fea2962bf: Status 404 returned error can't find the container with id dd5854a7dfbd63ad22102624c3a429062c53a69df4471e882d41b55fea2962bf
Jul 27 21:45:28.947832 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.947737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm" event={"ID":"fa074b02-4fda-479f-80b2-023cae3f7db8","Type":"ContainerStarted","Data":"895ea7364700e47ebdae2a4f1936f85fd8b83894e328f7b194df98dc332312bb"}
Jul 27 21:45:28.947832 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.947793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm" event={"ID":"fa074b02-4fda-479f-80b2-023cae3f7db8","Type":"ContainerStarted","Data":"dd5854a7dfbd63ad22102624c3a429062c53a69df4471e882d41b55fea2962bf"}
Jul 27 21:45:28.947832 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.947820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:28.963611 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:28.963566    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm" podStartSLOduration=1.9635512529999999 podStartE2EDuration="1.963551253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:28.963021346 +0000 UTC m=+3234.736953766" watchObservedRunningTime="2026-07-27 21:45:28.963551253 +0000 UTC m=+3234.737483692"
Jul 27 21:45:29.583744 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:29.583713    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-v6pj8_719c5e99-4b9e-4217-af88-fa2abfc025ef/download-server/0.log"
Jul 27 21:45:30.154384 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:30.154353    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-98m5s_e2c93b3d-1e00-489a-9682-f72c95ac215c/volume-data-source-validator/0.log"
Jul 27 21:45:31.255795 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:31.255763    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5dzsn_8b629b7e-680e-4019-a64e-0dc46331d1b9/dns/0.log"
Jul 27 21:45:31.279805 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:31.279772    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5dzsn_8b629b7e-680e-4019-a64e-0dc46331d1b9/kube-rbac-proxy/0.log"
Jul 27 21:45:31.378072 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:31.378031    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52wdx_43894920-1dc9-4687-ad29-f55409631161/dns-node-resolver/0.log"
Jul 27 21:45:31.937260 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:31.937228    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-659bfb4c76-tld5w_4257a616-d994-4f32-8cc9-6efa447aa1a1/registry/0.log"
Jul 27 21:45:31.986004 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:31.985965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jnmb6_bab66c4e-4a49-4752-9050-e2a53b6f9813/node-ca/0.log"
Jul 27 21:45:33.095030 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:33.095003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-n955m_03ae1502-2e08-4c91-8ce1-4db469bdbaed/kube-auth-proxy/0.log"
Jul 27 21:45:33.227568 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:33.227536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pxn4h_a9ffe201-d02b-4136-893d-656f80bb5e36/payload-pre-processing/0.log"
Jul 27 21:45:33.259100 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:33.259058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:45:33.282578 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:33.282548    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6b4bf9c-jn462_72c0faa1-cedb-4a19-838f-6e37a26255e3/router/0.log"
Jul 27 21:45:34.097111 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:34.097084    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n62fk_f4618ee4-fea8-4ac0-865f-8bf5c5c72c68/serve-healthcheck-canary/0.log"
Jul 27 21:45:34.655845 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:34.655814    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xpr8_d37b317e-a9ec-466f-ad1f-4c43d9ccff51/kube-rbac-proxy/0.log"
Jul 27 21:45:34.680192 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:34.680165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xpr8_d37b317e-a9ec-466f-ad1f-4c43d9ccff51/exporter/0.log"
Jul 27 21:45:34.706601 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:34.706560    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xpr8_d37b317e-a9ec-466f-ad1f-4c43d9ccff51/extractor/0.log"
Jul 27 21:45:34.963746 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:34.963718    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-xtlmm"
Jul 27 21:45:37.333601 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:37.333568    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55f67d7fff-6p9ql_783e4803-42ff-47e9-b2c8-25dafefec617/manager/0.log"
Jul 27 21:45:37.551544 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:37.551511    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-dqcxf_2d46802e-6d8d-41b1-86af-811c999d9b9e/manager/0.log"
Jul 27 21:45:37.577681 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:37.577652    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-dqcxf_2d46802e-6d8d-41b1-86af-811c999d9b9e/copy-manifests/0.log"
Jul 27 21:45:38.904116 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:38.904078    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-n955m_03ae1502-2e08-4c91-8ce1-4db469bdbaed/kube-auth-proxy/0.log"
Jul 27 21:45:39.009955 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:39.009921    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pxn4h_a9ffe201-d02b-4136-893d-656f80bb5e36/payload-pre-processing/0.log"
Jul 27 21:45:39.032817 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:39.032784    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:45:39.051330 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:39.051253    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6b4bf9c-jn462_72c0faa1-cedb-4a19-838f-6e37a26255e3/router/0.log"
Jul 27 21:45:39.058584 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:39.058524    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f9595876c-tw8ml_2697bfa4-8005-4ad7-b86e-cf62fc940703/manager/0.log"
Jul 27 21:45:40.582615 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:40.582581    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-n955m_03ae1502-2e08-4c91-8ce1-4db469bdbaed/kube-auth-proxy/0.log"
Jul 27 21:45:40.682363 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:40.682328    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pxn4h_a9ffe201-d02b-4136-893d-656f80bb5e36/payload-pre-processing/0.log"
Jul 27 21:45:40.705749 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:40.705714    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5jb2v_8deca1b0-401f-47eb-b698-36f931657c2c/payload-processing/0.log"
Jul 27 21:45:40.723958 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:40.723927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c6b4bf9c-jn462_72c0faa1-cedb-4a19-838f-6e37a26255e3/router/0.log"
Jul 27 21:45:44.148597 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:44.148572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9gsnq_047c7c11-68fb-426e-9ed3-56de63b6b1c7/migrator/0.log"
Jul 27 21:45:44.170354 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:44.170324    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9gsnq_047c7c11-68fb-426e-9ed3-56de63b6b1c7/graceful-termination/0.log"
Jul 27 21:45:45.769482 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.769452    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:45:45.795932 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.795901    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/egress-router-binary-copy/0.log"
Jul 27 21:45:45.819954 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.819926    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/cni-plugins/0.log"
Jul 27 21:45:45.842315 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.842245    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/bond-cni-plugin/0.log"
Jul 27 21:45:45.864558 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.864530    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/routeoverride-cni/0.log"
Jul 27 21:45:45.886871 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.886844    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/whereabouts-cni-bincopy/0.log"
Jul 27 21:45:45.911268 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:45.911239    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwpt7_10344088-6c8e-4058-bd0b-caf6897c208c/whereabouts-cni/0.log"
Jul 27 21:45:46.138754 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:46.138678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lwpmh_d3f50019-4b78-4178-b075-f78a0636d68b/kube-multus/0.log"
Jul 27 21:45:46.189981 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:46.189950    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fd7cv_3401dd96-76a0-49f2-8934-ceed9c91835c/network-metrics-daemon/0.log"
Jul 27 21:45:46.214346 ip-10-0-131-89 kubenswrapper[2580]: I0727 21:45:46.214322    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fd7cv_3401dd96-76a0-49f2-8934-ceed9c91835c/kube-rbac-proxy/0.log"