Jul 24 13:34:20.418806 ip-10-0-136-88 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 13:34:20.835082 ip-10-0-136-88 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:34:20.835082 ip-10-0-136-88 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 13:34:20.835082 ip-10-0-136-88 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:34:20.835082 ip-10-0-136-88 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 13:34:20.835082 ip-10-0-136-88 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 13:34:20.837576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.837404    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 13:34:20.842001 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.841980    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842005    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842009    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842013    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842016    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842019    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842022    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842025    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842027    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842030    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842033    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842035    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842038    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842040    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842043    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842045    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842048    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842051    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842054    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842056    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.842050 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842059    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842063    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842065    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842068    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842070    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842073    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842075    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842078    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842080    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842083    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842085    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842087    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842090    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842093    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842095    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842097    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842100    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842102    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842104    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842107    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.842591 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842110    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842112    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842114    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842117    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842119    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842122    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842124    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842126    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842129    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842131    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842133    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842136    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842139    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842141    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842145    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842149    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842151    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842154    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842157    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842159    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.843084 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842162    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842164    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842166    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842169    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842172    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842174    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842177    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842179    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842184    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842188    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842190    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842193    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842196    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842198    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842201    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842203    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842207    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842210    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842218    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.843593 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842222    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842225    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842228    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842231    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842234    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842236    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842239    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842242    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842244    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842247    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842250    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842252    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842255    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842257    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842260    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842262    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842265    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842267    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842270    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.842272    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.844053 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843042    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843050    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843055    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843059    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843063    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843067    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843070    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843073    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843076    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843079    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843082    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843086    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843089    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843092    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843094    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843097    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843100    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843103    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843106    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.844543 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843109    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843112    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843115    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843117    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843120    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843122    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843125    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843127    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843130    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843133    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843135    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843138    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843140    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843143    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843146    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843148    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843151    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843154    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843156    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843159    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.845018 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843161    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843163    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843166    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843169    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843171    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843174    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843178    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843181    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843184    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843187    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843189    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843192    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843195    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843198    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843200    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843203    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843205    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843208    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843210    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.845537 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843214    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843217    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843219    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843221    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843224    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843226    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843228    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843231    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843234    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843236    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843239    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843241    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843244    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843246    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843248    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843251    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843254    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843257    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843259    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843261    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.846003 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843264    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843266    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843269    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843271    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843274    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843276    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843279    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843281    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843284    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843286    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843289    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843291    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843293    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843296    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843300    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843302    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843304    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843307    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843309    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843311    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.846510 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843314    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843746    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843752    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843755    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843758    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843760    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843765    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843769    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843772    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843775    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843778    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843781    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843784    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843786    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843789    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843792    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843794    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843797    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843800    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.847009 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843803    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843805    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843807    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843810    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843812    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843815    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843819    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843822    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843824    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843827    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843829    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843832    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843834    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843836    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843839    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843842    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843844    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843847    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843849    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843851    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.847464 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843854    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843857    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843859    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843862    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843865    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843868    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843870    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843873    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843875    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843877    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843880    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843882    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843884    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843887    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843891    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843894    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843897    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843899    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843904    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843907    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.847995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843909    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843912    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843914    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843917    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843919    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843923    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843925    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843928    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843930    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843933    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843935    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843938    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843942    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843944    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843947    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843949    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843952    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843954    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843957    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843959    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.848492 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843962    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843964    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843967    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843969    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843971    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843974    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843977    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843979    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843981    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843984    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843989    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843991    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843994    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843996    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.843998    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844001    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844004    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844006    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844009    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844011    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.848981 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844014    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844092    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844100    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844107    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844113    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844117    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844121    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844126    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844130    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844133    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844136    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844140    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844144    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844147    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844150    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844153    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844156    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844158    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844162    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844167    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844170    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844173    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844177    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844180    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 13:34:20.849528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844184    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844187    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844190    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844194    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844197    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844200    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844203    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844206    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844209    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844214    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844217    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844220    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844224    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844227    2579 flags.go:64] FLAG: --enable-server="true"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844230    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844234    2579 flags.go:64] FLAG: --event-burst="100"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844237    2579 flags.go:64] FLAG: --event-qps="50"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844240    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844243    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844246    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844249    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844252    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844255    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844258    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844261    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 13:34:20.850090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844264    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844267    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844270    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844273    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844276    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844280    2579 flags.go:64] FLAG: --feature-gates=""
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844284    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844287    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844290    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844293    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844296    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844299    2579 flags.go:64] FLAG: --help="false"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844302    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-88.ec2.internal"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844305    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844308    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844311    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844315    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844318    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844321    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844325    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844328    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844331    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844334    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844337    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 13:34:20.850814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844340    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844343    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844346    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844349    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844351    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844354    2579 flags.go:64] FLAG: --lock-file=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844357    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844360    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844363    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844368    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844371    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844373    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844376    2579 flags.go:64] FLAG: --logging-format="text"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844380    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844384    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844387    2579 flags.go:64] FLAG: --manifest-url=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844389    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844393    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844396    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844400    2579 flags.go:64] FLAG: --max-pods="110"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844404    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844407    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844410    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844413    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844416    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 13:34:20.851742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844418    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844429    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844442    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844446    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844449    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844452    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844455    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844460    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844463    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844466    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844468    2579 flags.go:64] FLAG: --port="10250"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844472    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844488    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1285c342c83ffb8"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844491    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844494    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844497    2579 flags.go:64] FLAG: --register-node="true"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844500    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844503    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844506    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844509    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844513    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844517    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844520    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844523    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844526    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 13:34:20.852703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844529    2579 flags.go:64] FLAG: --runonce="false"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844531    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844535    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844538    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844540    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844543    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844546    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844550    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844553    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844556    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844561    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844564    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844567    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844570    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844573    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844576    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844581    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844584    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844586    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844590    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844593    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844596    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844599    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844602    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844605    2579 flags.go:64] FLAG: --v="2"
Jul 24 13:34:20.853329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844608    2579 flags.go:64] FLAG: --version="false"
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844612    2579 flags.go:64] FLAG: --vmodule=""
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844617    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.844621    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844759    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844763    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844766    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844769    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844771    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844774    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844776    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844779    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844782    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844784    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844787    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844790    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844792    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844797    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844799    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844802    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.853961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844805    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844807    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844810    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844812    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844815    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844817    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844820    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844822    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844825    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844829    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844832    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844835    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844838    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844841    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844848    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844851    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844854    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844857    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844859    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844862    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.854442 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844864    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844867    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844870    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844872    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844875    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844877    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844880    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844882    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844885    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844888    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844892    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844894    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844897    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844900    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844902    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844905    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844907    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844909    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844912    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844914    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.854999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844917    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844919    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844922    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844924    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844927    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844929    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844933    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844935    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844938    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844940    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844943    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844945    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844948    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844950    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844953    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844955    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844957    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844961    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844964    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.855506 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844967    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844969    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844973    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844978    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844981    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844984    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844987    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844990    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844992    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844995    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.844997    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845000    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845002    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845005    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845008    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845010    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845012    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845015    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845018    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.855961 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845022    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845024    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845027    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845029    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.845032    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.845042    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.853458    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.853498    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.854314    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.854323    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854397    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854403    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854406    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854410    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.856422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854413    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854416    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854419    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854422    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854424    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854427    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854429    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854432    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854435    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854437    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854440    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854443    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854446    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854448    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854451    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854453    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854456    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854458    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854461    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854464    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.856795 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854466    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854469    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854471    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854495    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854499    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854503    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854506    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854509    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854512    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854514    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854517    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854519    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854522    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854525    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854528    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854530    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854532    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854535    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854538    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854540    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.857296 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854543    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854546    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854549    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854551    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854554    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854556    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854559    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854561    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854564    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854566    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854568    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854571    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854573    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854576    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854579    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854582    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854584    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854587    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854590    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854593    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.857806 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854595    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854598    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854601    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854603    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854605    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854610    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854613    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854616    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854619    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854621    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854624    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854628    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854631    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854634    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854637    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854640    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854643    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854646    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854648    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.858378 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854651    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854653    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854657    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854660    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854663    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854666    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854668    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854671    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854673    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854676    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854679    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854682    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854684    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854687    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854690    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.859005 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854692    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.854698    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854833    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854838    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854841    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854844    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854847    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854850    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854853    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854865    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854868    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854871    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854874    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854876    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854880    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 13:34:20.859495 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854883    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854885    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854888    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854891    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854893    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854896    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854898    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854901    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854903    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854906    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854909    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854912    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854915    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854917    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854920    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854923    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854925    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854928    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854930    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 13:34:20.859871 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854932    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854935    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854937    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854940    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854945    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854947    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854949    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854952    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854954    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854957    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854959    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854963    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854966    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854969    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854971    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854976    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854979    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854981    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 13:34:20.860395 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854983    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854986    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854988    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854991    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854994    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854996    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.854999    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855002    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855004    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855007    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855009    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855012    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855015    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855017    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855020    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855024    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855027    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855030    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855033    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 13:34:20.860935 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855035    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855038    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855041    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855043    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855046    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855048    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855051    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855055    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855058    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855061    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855064    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855067    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855070    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855072    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855075    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855077    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855080    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855083    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855085    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 13:34:20.861722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855088    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855090    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855093    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855095    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855098    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855100    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855103    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855105    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:20.855108    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.855113    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.855285    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.857875    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.859138    2579 server.go:1019] "Starting client certificate rotation"
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.859268    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:34:20.862412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.859320    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 13:34:20.862786 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.861546    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:34:20.863297 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.863265    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 13:34:20.873848 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.873830    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 24 13:34:20.877023 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.877008    2579 log.go:25] "Validated CRI v1 image API"
Jul 24 13:34:20.877865 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.877847    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 13:34:20.880328 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.880304    2579 fs.go:135] Filesystem UUIDs: map[02d58f7a-320e-4c75-a17f-b29ddf9374d2:/dev/nvme0n1p3 718e448d-9371-4799-9173-4d14e72f7ebc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 24 13:34:20.880402 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.880326    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 13:34:20.888065 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.887930    2579 manager.go:216] Machine: {Timestamp:2026-07-24 13:34:20.886286015 +0000 UTC m=+0.348892851 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088998 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd37efdbf5b0fdd96d698331f3349 SystemUUID:ec2bd37e-fdbf-5b0f-dd96-d698331f3349 BootID:cd34ac38-d927-4ea2-858c-f4e753f9abcf Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:41:da:75:25:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:41:da:75:25:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:17:14:9d:f0:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 13:34:20.888065 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.888050    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 13:34:20.888255 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.888177    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 13:34:20.891177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.891146    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 13:34:20.891350 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.891177    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 13:34:20.891854 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.891841    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 13:34:20.891919 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.891858    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 13:34:20.891919 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.891869    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:34:20.892764 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.892751    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 13:34:20.893549 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.893537    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 13:34:20.895448 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.895433    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:34:20.895608 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.895596    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 13:34:20.900861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.900848    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 13:34:20.900921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.900868    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 13:34:20.900921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.900897    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 13:34:20.900921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.900911    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 24 13:34:20.901062 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.900924    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 13:34:20.901391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.901377    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:34:20.901454 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.901401    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 13:34:20.903939 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.903920    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 13:34:20.905744 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.905727    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 13:34:20.905798 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.905772    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 13:34:20.907593 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907570    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 13:34:20.907593 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907596    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907606    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907613    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907622    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907627    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907635    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907639    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907646    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907652    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907660    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 13:34:20.907702 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.907666    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 13:34:20.908492 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.908459    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 13:34:20.908492 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.908470    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 13:34:20.911644 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.911631    2579 server.go:1268] "Started kubelet"
Jul 24 13:34:20.911827 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.911769    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 13:34:20.911896 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.911883    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 13:34:20.912081 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.911932    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 13:34:20.912614 ip-10-0-136-88 systemd[1]: Started Kubernetes Kubelet.
Jul 24 13:34:20.913818 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.913790    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 13:34:20.914917 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.914902    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 13:34:20.919554 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.919535    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 13:34:20.919642 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.919554    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 13:34:20.919688 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.919645    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:20.920112 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.920095    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 13:34:20.920188 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.920120    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 13:34:20.920188 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.920095    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 13:34:20.920313 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.920297    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 13:34:20.920313 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.920311    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 13:34:20.921068 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921051    2579 factory.go:55] Registering systemd factory
Jul 24 13:34:20.921068 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921069    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 24 13:34:20.921191 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921094    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 13:34:20.921191 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.921093    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 13:34:20.921191 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.921180    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 13:34:20.921405 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921368    2579 factory.go:153] Registering CRI-O factory
Jul 24 13:34:20.921405 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921383    2579 factory.go:223] Registration of the crio container factory successfully
Jul 24 13:34:20.921544 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.920358    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260a358a96  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.911602326 +0000 UTC m=+0.374209162,LastTimestamp:2026-07-24 13:34:20.911602326 +0000 UTC m=+0.374209162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.921544 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921435    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 13:34:20.921544 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921468    2579 factory.go:103] Registering Raw factory
Jul 24 13:34:20.921714 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.921584    2579 manager.go:1196] Started watching for new ooms in manager
Jul 24 13:34:20.922237 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.922219    2579 manager.go:349] Starting recovery of all containers
Jul 24 13:34:20.925846 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.925807    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 13:34:20.925993 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.925968    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 13:34:20.927685 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.927660    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 13:34:20.935079 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.934937    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:34:20.935661 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.935641    2579 manager.go:354] Recovery completed
Jul 24 13:34:20.940227 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.940212    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:20.944185 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944149    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:20.944254 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944208    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:20.944254 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944226    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:20.944823 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944804    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 13:34:20.944823 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944821    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 13:34:20.944945 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944837    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 13:34:20.944945 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.944813    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:20.945697 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.945682    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:20.945755 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.945709    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:20.945755 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.945719    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:20.946358 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.946347    2579 policy_none.go:49] "None policy: Start"
Jul 24 13:34:20.946394 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.946363    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 13:34:20.946394 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.946374    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 13:34:20.947080 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.947007    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260c26c055  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.944187477 +0000 UTC m=+0.406794319,LastTimestamp:2026-07-24 13:34:20.944187477 +0000 UTC m=+0.406794319,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.947544 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.947528    2579 policy_none.go:47] "Start"
Jul 24 13:34:20.956728 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.956654    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260c273175  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.944216437 +0000 UTC m=+0.406823275,LastTimestamp:2026-07-24 13:34:20.944216437 +0000 UTC m=+0.406823275,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.962382 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.962293    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260c2770b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-88.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.944232624 +0000 UTC m=+0.406839460,LastTimestamp:2026-07-24 13:34:20.944232624 +0000 UTC m=+0.406839460,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.968447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.968425    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhbvk"
Jul 24 13:34:20.968572 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.968469    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-88.ec2.internal.18c53d260c26c055\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260c26c055  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.944187477 +0000 UTC m=+0.406794319,LastTimestamp:2026-07-24 13:34:20.945696524 +0000 UTC m=+0.408303358,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.978196 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.978127    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-88.ec2.internal.18c53d260c273175\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-88.ec2.internal.18c53d260c273175  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-88.ec2.internal,UID:ip-10-0-136-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-88.ec2.internal,},FirstTimestamp:2026-07-24 13:34:20.944216437 +0000 UTC m=+0.406823275,LastTimestamp:2026-07-24 13:34:20.945713709 +0000 UTC m=+0.408320547,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-88.ec2.internal,}"
Jul 24 13:34:20.978711 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.978693    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhbvk"
Jul 24 13:34:20.996150 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996132    2579 manager.go:337] "Starting Device Plugin manager"
Jul 24 13:34:20.996265 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.996167    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 13:34:20.996265 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996176    2579 server.go:87] "Starting device plugin registration server"
Jul 24 13:34:20.996685 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996672    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 13:34:20.996762 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996686    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 13:34:20.996823 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996791    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 13:34:20.996939 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996872    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 13:34:20.996939 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:20.996886    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 13:34:20.997441 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.997423    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 13:34:20.997565 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:20.997490    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.096847 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.096768    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:21.097823 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.097796    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:21.097901 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.097832    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:21.097901 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.097850    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:21.097901 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.097894    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.104410 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.104390    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.104511 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.104416    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-88.ec2.internal\": node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.116530 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.116496    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 13:34:21.118011 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.117984    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 13:34:21.118129 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.118028    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 13:34:21.118129 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.118053    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 13:34:21.118129 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.118095    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 13:34:21.120609 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.120585    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:21.125565 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.125544    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.219192 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.219148    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"]
Jul 24 13:34:21.219310 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.219269    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:21.221861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.221841    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:21.221969 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.221876    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:21.221969 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.221887    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:21.222187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.222168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.222222 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.222196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.223053 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223040    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:21.223147 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.223202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223176    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:21.223853 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223838    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:21.223914 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223864    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:21.223914 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223878    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:21.223914 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223895    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:21.224043 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223918    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:21.224043 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.223933    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:21.225176 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.225162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.225235 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.225190    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:21.225652 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.225633    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.225884 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.225869    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:21.225947 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.225901    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:21.225947 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.225916    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:21.248197 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.248173    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-88.ec2.internal\" not found" node="ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.252680 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.252663    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-88.ec2.internal\" not found" node="ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.323034 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.323006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86bdcc87ec35dd1e444c5a9b19ae75f0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"86bdcc87ec35dd1e444c5a9b19ae75f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.323230 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.323053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.323230 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.323128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.323230 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.323151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.323230 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.323202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a18f901a3f872b6b7b15cb8e678eac3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal\" (UID: \"1a18f901a3f872b6b7b15cb8e678eac3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.326430 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.326415    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.423715 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.423579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86bdcc87ec35dd1e444c5a9b19ae75f0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"86bdcc87ec35dd1e444c5a9b19ae75f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.423715 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.423610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86bdcc87ec35dd1e444c5a9b19ae75f0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-88.ec2.internal\" (UID: \"86bdcc87ec35dd1e444c5a9b19ae75f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.426707 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.426680    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.527324 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.527291    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.550517 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.550463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.556607 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.556586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:21.628008 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.627966    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.728460 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.728428    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.828835 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.828798    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.858989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.858949    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 13:34:21.859465 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.859133    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 13:34:21.929842 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:21.929812    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:21.983079 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.982987    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 13:29:20 +0000 UTC" deadline="2028-04-25 14:27:52.568439776 +0000 UTC"
Jul 24 13:34:21.983079 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:21.983024    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15384h53m30.585419239s"
Jul 24 13:34:22.009312 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.009284    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 13:34:22.010303 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.010287    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 13:34:22.010398 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.010316    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 13:34:22.010398 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.010328    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeHasSufficientPID"
Jul 24 13:34:22.030492 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:22.030444    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:22.130980 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:22.130950    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-88.ec2.internal\" not found"
Jul 24 13:34:22.173192 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:22.173152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bdcc87ec35dd1e444c5a9b19ae75f0.slice/crio-f86a0b146d01759cdc8314a103ddf36206ee928b2995e5f9b0b95c74d84fede5 WatchSource:0}: Error finding container f86a0b146d01759cdc8314a103ddf36206ee928b2995e5f9b0b95c74d84fede5: Status 404 returned error can't find the container with id f86a0b146d01759cdc8314a103ddf36206ee928b2995e5f9b0b95c74d84fede5
Jul 24 13:34:22.176228 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.176212    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:22.179536 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.179521    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:34:22.209165 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:22.209145    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a18f901a3f872b6b7b15cb8e678eac3.slice/crio-2e29637ab7e17da169367d7357ff69eaa7297f66bf994fd09d3e13bd249eeb06 WatchSource:0}: Error finding container 2e29637ab7e17da169367d7357ff69eaa7297f66bf994fd09d3e13bd249eeb06: Status 404 returned error can't find the container with id 2e29637ab7e17da169367d7357ff69eaa7297f66bf994fd09d3e13bd249eeb06
Jul 24 13:34:22.220746 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.220728    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:22.235847 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.235801    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:34:22.238466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.238441    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal"
Jul 24 13:34:22.247435 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.247417    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 13:34:22.283251 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.283226    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:22.466735 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.466705    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:22.902074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.902039    2579 apiserver.go:52] "Watching apiserver"
Jul 24 13:34:22.914712 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.914683    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 13:34:22.915816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.915788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal","openshift-multus/multus-additional-cni-plugins-6f5pk","openshift-network-diagnostics/network-check-target-djh8n","kube-system/konnectivity-agent-s6nth","kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal","openshift-image-registry/node-ca-7qndb","openshift-multus/multus-h68kb","openshift-multus/network-metrics-daemon-77ztq","openshift-network-operator/iptables-alerter-gqjsc","openshift-ovn-kubernetes/ovnkube-node-bbhmq","kube-system/global-pull-secret-syncer-j8frr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj","openshift-cluster-node-tuning-operator/tuned-lp8w5","openshift-dns/node-resolver-dwt4j"]
Jul 24 13:34:22.918819 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.918159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.920240 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.920221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:22.920344 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:22.920289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:22.920469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.920455    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 13:34:22.921619 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.921601    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lvsn9\""
Jul 24 13:34:22.921719 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.921700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.921777 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.921700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.921827 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.921781    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 13:34:22.921884 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.921838    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 13:34:22.922301 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.922282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:22.922378 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:22.922346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:22.924671 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.924650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:22.927163 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.927144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:22.927611 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.927584    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 13:34:22.927964 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.927948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5gvh6\""
Jul 24 13:34:22.928614 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.928595    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 13:34:22.930216 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.929227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:22.930216 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.929426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.930216 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.929669    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fb8xr\""
Jul 24 13:34:22.930216 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.930051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzj5\" (UniqueName: \"kubernetes.io/projected/f9088c09-c41e-4ee9-aa85-79c17a53b630-kube-api-access-bmzj5\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-cni-binary-copy\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-socket-dir-parent\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-netns\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-multus-certs\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-dbus\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-system-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-cnibin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-os-release\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-bin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-conf-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-daemon-config\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:22.931391 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-k8s-cni-cncf-io\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-multus\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-hostroot\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-etc-kubernetes\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-kubelet-config\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-kubelet\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:22.932187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.931551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.933031    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.933321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5vfnj\""
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.933694    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.933952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.934216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 13:34:22.935445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.934241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 13:34:22.937070 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:22.937163 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.937163 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937144    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.937716 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qh6zf\""
Jul 24 13:34:22.937884 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.937973 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.937913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cqxlw\""
Jul 24 13:34:22.938120 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.938106    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.939554 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.939534    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:22.940198 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.940180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 13:34:22.940749 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.940618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 13:34:22.940749 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.940668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zwwtv\""
Jul 24 13:34:22.942503 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.942139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:22.942503 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:22.942203    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:22.944301 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.944284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:22.947847 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.947816    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 13:34:22.947940 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.947899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 13:34:22.947940 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.947818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-27l9r\""
Jul 24 13:34:22.948045 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.947950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 13:34:22.948110 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.948536 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xpt82\""
Jul 24 13:34:22.948630 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948566    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.948630 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948588    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 13:34:22.948740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948701    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 13:34:22.948812 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948794    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 13:34:22.948877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.948865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 13:34:22.954835 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:22.954814    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 13:34:23.000779 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.000749    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5hg5"
Jul 24 13:34:23.010406 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.010350    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5hg5"
Jul 24 13:34:23.012281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.012265    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 13:34:23.021301 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.021282    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 13:34:23.032708 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:23.032806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21c3ce53-0fde-450c-b0f0-28920045b672-host\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.032806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-systemd-units\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.032806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-cnibin\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.032913 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-hosts-file\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.032913 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-modprobe-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.032985 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.033030 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.032980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-socket-dir-parent\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033083 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033083 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-socket-dir-parent\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033184 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-config\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033184 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-lib-modules\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.033184 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.033332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-log-socket\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-kubernetes\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.033332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-kubelet-config\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.033332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-var-lib-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-script-lib\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdcj\" (UniqueName: \"kubernetes.io/projected/fee69372-6d61-41d8-bed4-b45a4200af87-kube-api-access-9kdcj\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-kubelet-config\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzj5\" (UniqueName: \"kubernetes.io/projected/f9088c09-c41e-4ee9-aa85-79c17a53b630-kube-api-access-bmzj5\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-netns\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-cni-binary-copy\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-netns\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-multus-certs\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-ovn\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-device-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-multus-certs\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-netns\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-slash\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-systemd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-system-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-bin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.033861 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-system-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-socket-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-bin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-k8s-cni-cncf-io\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.033991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-hostroot\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-run-k8s-cni-cncf-io\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-kubelet\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-netd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-cni-binary-copy\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngq4j\" (UniqueName: \"kubernetes.io/projected/0b1f159e-12a1-418a-97d6-8de1de5fa880-kube-api-access-ngq4j\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-hostroot\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vp5\" (UniqueName: \"kubernetes.io/projected/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-kube-api-access-j6vp5\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-etc-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-env-overrides\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovn-node-metrics-cert\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9486w\" (UniqueName: \"kubernetes.io/projected/687056e0-a635-4b5f-b0b2-62588ad14d67-kube-api-access-9486w\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-conf\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d681c4f4-490d-4200-8220-72d44766fa11-agent-certs\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-dbus\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7m5d\" (UniqueName: \"kubernetes.io/projected/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-kube-api-access-q7m5d\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2kw\" (UniqueName: \"kubernetes.io/projected/21c3ce53-0fde-450c-b0f0-28920045b672-kube-api-access-7n2kw\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-dbus\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-system-cni-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.034668    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d681c4f4-490d-4200-8220-72d44766fa11-konnectivity-ca\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.034759    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:23.534718411 +0000 UTC m=+2.997325233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.034938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-node-log\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sks65\" (UniqueName: \"kubernetes.io/projected/736a8f69-fccf-46e0-9731-d5447ca9317c-kube-api-access-sks65\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-registration-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-multus\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-cni-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-cni-multus\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-etc-kubernetes\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-etc-kubernetes\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.034970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/687056e0-a635-4b5f-b0b2-62588ad14d67-iptables-alerter-script\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysconfig\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-run\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-host\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-os-release\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-kubelet\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-tmp-dir\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-systemd\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.035576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-sys\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-var-lib-kubelet\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-host-var-lib-kubelet\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-etc-tuned\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-tmp\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21c3ce53-0fde-450c-b0f0-28920045b672-serviceca\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687056e0-a635-4b5f-b0b2-62588ad14d67-host-slash\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-sys-fs\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxltm\" (UniqueName: \"kubernetes.io/projected/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kube-api-access-fxltm\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-bin\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-cnibin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-os-release\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-conf-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-cnibin\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-daemon-config\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-conf-dir\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036649 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f9088c09-c41e-4ee9-aa85-79c17a53b630-os-release\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.036649 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.035833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f9088c09-c41e-4ee9-aa85-79c17a53b630-multus-daemon-config\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.042183 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.042155    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:23.042183 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.042186    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:23.042375 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.042201    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:23.042375 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.042269    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:23.542250933 +0000 UTC m=+3.004857766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:23.042375 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.042349    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 13:34:23.046727 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.046703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzj5\" (UniqueName: \"kubernetes.io/projected/f9088c09-c41e-4ee9-aa85-79c17a53b630-kube-api-access-bmzj5\") pod \"multus-h68kb\" (UID: \"f9088c09-c41e-4ee9-aa85-79c17a53b630\") " pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.122820 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.122768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"1a18f901a3f872b6b7b15cb8e678eac3","Type":"ContainerStarted","Data":"2e29637ab7e17da169367d7357ff69eaa7297f66bf994fd09d3e13bd249eeb06"}
Jul 24 13:34:23.123880 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.123848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" event={"ID":"86bdcc87ec35dd1e444c5a9b19ae75f0","Type":"ContainerStarted","Data":"f86a0b146d01759cdc8314a103ddf36206ee928b2995e5f9b0b95c74d84fede5"}
Jul 24 13:34:23.135778 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-kubelet\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.135778 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-netd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngq4j\" (UniqueName: \"kubernetes.io/projected/0b1f159e-12a1-418a-97d6-8de1de5fa880-kube-api-access-ngq4j\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vp5\" (UniqueName: \"kubernetes.io/projected/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-kube-api-access-j6vp5\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-kubelet\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-netd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.135962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.135949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.136195 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-etc-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136195 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136195 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-etc-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136195 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136195 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-env-overrides\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136396 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovn-node-metrics-cert\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136396 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9486w\" (UniqueName: \"kubernetes.io/projected/687056e0-a635-4b5f-b0b2-62588ad14d67-kube-api-access-9486w\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.136467 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-conf\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.136467 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d681c4f4-490d-4200-8220-72d44766fa11-agent-certs\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7m5d\" (UniqueName: \"kubernetes.io/projected/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-kube-api-access-q7m5d\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2kw\" (UniqueName: \"kubernetes.io/projected/21c3ce53-0fde-450c-b0f0-28920045b672-kube-api-access-7n2kw\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-system-cni-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-conf\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.136592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d681c4f4-490d-4200-8220-72d44766fa11-konnectivity-ca\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-node-log\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sks65\" (UniqueName: \"kubernetes.io/projected/736a8f69-fccf-46e0-9731-d5447ca9317c-kube-api-access-sks65\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-registration-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/687056e0-a635-4b5f-b0b2-62588ad14d67-iptables-alerter-script\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-env-overrides\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysconfig\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-system-cni-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-run\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-host\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-registration-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-os-release\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-tmp-dir\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.136864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-node-log\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysconfig\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-run\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-systemd\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-host\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.136968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-sys\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.137024    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-var-lib-kubelet\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-etc-tuned\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-systemd\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-tmp\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-sys\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d681c4f4-490d-4200-8220-72d44766fa11-konnectivity-ca\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21c3ce53-0fde-450c-b0f0-28920045b672-serviceca\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687056e0-a635-4b5f-b0b2-62588ad14d67-host-slash\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.137576 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-tmp-dir\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.137323    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:23.637304602 +0000 UTC m=+3.099911433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-sys-fs\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137400    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxltm\" (UniqueName: \"kubernetes.io/projected/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kube-api-access-fxltm\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-bin\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/687056e0-a635-4b5f-b0b2-62588ad14d67-iptables-alerter-script\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/687056e0-a635-4b5f-b0b2-62588ad14d67-host-slash\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-sys-fs\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-os-release\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-cni-bin\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-var-lib-kubelet\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21c3ce53-0fde-450c-b0f0-28920045b672-host\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-systemd-units\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21c3ce53-0fde-450c-b0f0-28920045b672-host\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.138360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-cnibin\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-cnibin\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21c3ce53-0fde-450c-b0f0-28920045b672-serviceca\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.137993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-systemd-units\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-hosts-file\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-modprobe-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-hosts-file\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-modprobe-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-config\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-lib-modules\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-log-socket\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-log-socket\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.139141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-kubernetes\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-var-lib-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/736a8f69-fccf-46e0-9731-d5447ca9317c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-kubernetes\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-script-lib\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-var-lib-openvswitch\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdcj\" (UniqueName: \"kubernetes.io/projected/fee69372-6d61-41d8-bed4-b45a4200af87-kube-api-access-9kdcj\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-lib-modules\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-config\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-netns\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-ovn\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-run-netns\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-ovn\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.138972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-device-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-slash\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-systemd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-socket-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fee69372-6d61-41d8-bed4-b45a4200af87-etc-sysctl-d\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-host-slash\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovnkube-script-lib\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-device-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-socket-dir\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b1f159e-12a1-418a-97d6-8de1de5fa880-run-systemd\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/736a8f69-fccf-46e0-9731-d5447ca9317c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.139782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b1f159e-12a1-418a-97d6-8de1de5fa880-ovn-node-metrics-cert\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.140977 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.140216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-tmp\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.141632 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.141614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fee69372-6d61-41d8-bed4-b45a4200af87-etc-tuned\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.141931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.141912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d681c4f4-490d-4200-8220-72d44766fa11-agent-certs\") pod \"konnectivity-agent-s6nth\" (UID: \"d681c4f4-490d-4200-8220-72d44766fa11\") " pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.150777 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.150753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngq4j\" (UniqueName: \"kubernetes.io/projected/0b1f159e-12a1-418a-97d6-8de1de5fa880-kube-api-access-ngq4j\") pod \"ovnkube-node-bbhmq\" (UID: \"0b1f159e-12a1-418a-97d6-8de1de5fa880\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.154991 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.154923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vp5\" (UniqueName: \"kubernetes.io/projected/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-kube-api-access-j6vp5\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.156140 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.156114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7m5d\" (UniqueName: \"kubernetes.io/projected/fd7ee460-ed31-4b90-97f6-39b857c5dcf2-kube-api-access-q7m5d\") pod \"node-resolver-dwt4j\" (UID: \"fd7ee460-ed31-4b90-97f6-39b857c5dcf2\") " pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.156720 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.156675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxltm\" (UniqueName: \"kubernetes.io/projected/8db0d2ee-01ba-4838-95de-fc0ee3dff5c2-kube-api-access-fxltm\") pod \"aws-ebs-csi-driver-node-fvtwj\" (UID: \"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.158194 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.158173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sks65\" (UniqueName: \"kubernetes.io/projected/736a8f69-fccf-46e0-9731-d5447ca9317c-kube-api-access-sks65\") pod \"multus-additional-cni-plugins-6f5pk\" (UID: \"736a8f69-fccf-46e0-9731-d5447ca9317c\") " pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.158509 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.158469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9486w\" (UniqueName: \"kubernetes.io/projected/687056e0-a635-4b5f-b0b2-62588ad14d67-kube-api-access-9486w\") pod \"iptables-alerter-gqjsc\" (UID: \"687056e0-a635-4b5f-b0b2-62588ad14d67\") " pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.158773 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.158753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdcj\" (UniqueName: \"kubernetes.io/projected/fee69372-6d61-41d8-bed4-b45a4200af87-kube-api-access-9kdcj\") pod \"tuned-lp8w5\" (UID: \"fee69372-6d61-41d8-bed4-b45a4200af87\") " pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.158962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.158943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2kw\" (UniqueName: \"kubernetes.io/projected/21c3ce53-0fde-450c-b0f0-28920045b672-kube-api-access-7n2kw\") pod \"node-ca-7qndb\" (UID: \"21c3ce53-0fde-450c-b0f0-28920045b672\") " pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.232276 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.232239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h68kb"
Jul 24 13:34:23.238953 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.238929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:23.247904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.247870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dwt4j"
Jul 24 13:34:23.254559 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.254541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7qndb"
Jul 24 13:34:23.262200 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.262179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqjsc"
Jul 24 13:34:23.269832 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.269812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lp8w5"
Jul 24 13:34:23.277389 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.277368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6f5pk"
Jul 24 13:34:23.285162 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.285141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:23.290834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.290815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj"
Jul 24 13:34:23.542264 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.542177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:23.542412 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.542343    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:23.542470 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.542426    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:24.542400184 +0000 UTC m=+4.005007012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:23.642820 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.642782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:23.643005 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:23.642963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:23.643005 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.642978    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:23.643005 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.643004    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:23.643174 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.643017    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:23.643174 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.643094    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:23.643174 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.643114    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:24.643093987 +0000 UTC m=+4.105700832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:23.643174 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:23.643149    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:24.643132926 +0000 UTC m=+4.105739749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:24.013596 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.013551    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:29:23 +0000 UTC" deadline="2027-12-21 07:15:26.634045959 +0000 UTC"
Jul 24 13:34:24.013596 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.013590    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12353h41m2.62046126s"
Jul 24 13:34:24.122713 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:24.122688    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db0d2ee_01ba_4838_95de_fc0ee3dff5c2.slice/crio-fa48aa0c2ff4f779dda6b02e2a75a9cb4fec3d809e6c20a6756ae02c3d1067a0 WatchSource:0}: Error finding container fa48aa0c2ff4f779dda6b02e2a75a9cb4fec3d809e6c20a6756ae02c3d1067a0: Status 404 returned error can't find the container with id fa48aa0c2ff4f779dda6b02e2a75a9cb4fec3d809e6c20a6756ae02c3d1067a0
Jul 24 13:34:24.124194 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:24.124155    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7ee460_ed31_4b90_97f6_39b857c5dcf2.slice/crio-dbad0090e460e752f0e22503b2667229c03fa865b7d0c25a4536c1bea4bbfe79 WatchSource:0}: Error finding container dbad0090e460e752f0e22503b2667229c03fa865b7d0c25a4536c1bea4bbfe79: Status 404 returned error can't find the container with id dbad0090e460e752f0e22503b2667229c03fa865b7d0c25a4536c1bea4bbfe79
Jul 24 13:34:24.135575 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:34:24.131354    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f159e_12a1_418a_97d6_8de1de5fa880.slice/crio-2f2a30ff47243872b898cff552637f0c49e29d289054a348f4ad7f9437efd4c8 WatchSource:0}: Error finding container 2f2a30ff47243872b898cff552637f0c49e29d289054a348f4ad7f9437efd4c8: Status 404 returned error can't find the container with id 2f2a30ff47243872b898cff552637f0c49e29d289054a348f4ad7f9437efd4c8
Jul 24 13:34:24.135575 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.133659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj" event={"ID":"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2","Type":"ContainerStarted","Data":"fa48aa0c2ff4f779dda6b02e2a75a9cb4fec3d809e6c20a6756ae02c3d1067a0"}
Jul 24 13:34:24.549034 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.548653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:24.549189 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.548830    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:24.549189 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.549125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:26.549097523 +0000 UTC m=+6.011704353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.649740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:24.649804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.649964    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.649985    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.649998    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.650056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:26.650037148 +0000 UTC m=+6.112643974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.650470    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:24.650583 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:24.650542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:26.650525601 +0000 UTC m=+6.113132427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:25.014401 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.014286    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:29:23 +0000 UTC" deadline="2028-01-02 08:32:08.018246001 +0000 UTC"
Jul 24 13:34:25.014401 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.014349    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h57m43.003901685s"
Jul 24 13:34:25.118353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.118314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:25.118565 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:25.118447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:25.118653 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.118571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:25.118703 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:25.118666    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:25.118771 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.118731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:25.118820 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:25.118809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:25.156983 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.156943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"2f2a30ff47243872b898cff552637f0c49e29d289054a348f4ad7f9437efd4c8"}
Jul 24 13:34:25.180447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.179647    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a18f901a3f872b6b7b15cb8e678eac3" containerID="f9fe1a202c5a71f178678adcfdf9f4c0074e37863cc19abb0b452442a540ab4e" exitCode=0
Jul 24 13:34:25.180447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.179740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"1a18f901a3f872b6b7b15cb8e678eac3","Type":"ContainerDied","Data":"f9fe1a202c5a71f178678adcfdf9f4c0074e37863cc19abb0b452442a540ab4e"}
Jul 24 13:34:25.190095 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.190061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqjsc" event={"ID":"687056e0-a635-4b5f-b0b2-62588ad14d67","Type":"ContainerStarted","Data":"536af95730ef760215848100c4fc41dab06c0d1176d0264832326f20fc693900"}
Jul 24 13:34:25.203430 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.203292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwt4j" event={"ID":"fd7ee460-ed31-4b90-97f6-39b857c5dcf2","Type":"ContainerStarted","Data":"dbad0090e460e752f0e22503b2667229c03fa865b7d0c25a4536c1bea4bbfe79"}
Jul 24 13:34:25.220306 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.220264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" event={"ID":"86bdcc87ec35dd1e444c5a9b19ae75f0","Type":"ContainerStarted","Data":"94d863431b00d6d90b261d93c74a7c443ccb30a500915897917c1989e80551f0"}
Jul 24 13:34:25.228969 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.228933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lp8w5" event={"ID":"fee69372-6d61-41d8-bed4-b45a4200af87","Type":"ContainerStarted","Data":"74b7140bce17da455c2465439ecdd4f42e82ac8e1516b936619d02dcabd58723"}
Jul 24 13:34:25.234018 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.233860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerStarted","Data":"fb6628ce0cd3d4a965b42ff6c3f42336fddd6a1577dda4063d9645cfb9fe9d1d"}
Jul 24 13:34:25.253426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.253385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h68kb" event={"ID":"f9088c09-c41e-4ee9-aa85-79c17a53b630","Type":"ContainerStarted","Data":"49eb69ebd4e3b56d4dc16d2e7cb6c54bc1a80ae84c42c3c080ad330c4074294d"}
Jul 24 13:34:25.277437 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.277345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qndb" event={"ID":"21c3ce53-0fde-450c-b0f0-28920045b672","Type":"ContainerStarted","Data":"fbc4807c382ca57ba18213bb98b943e21dcbeaf323c5691f28f1193e301f68f3"}
Jul 24 13:34:25.299892 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:25.299852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6nth" event={"ID":"d681c4f4-490d-4200-8220-72d44766fa11","Type":"ContainerStarted","Data":"624be0c426d47220ae8f67d96c89b96e035d45710fafa390311f60c23e66164a"}
Jul 24 13:34:26.314426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.314393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" event={"ID":"1a18f901a3f872b6b7b15cb8e678eac3","Type":"ContainerStarted","Data":"a64beb0b20251745a1d9362001af7063d5bdb8218bbfa85b273efe6d65b63927"}
Jul 24 13:34:26.331952 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.331881    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-88.ec2.internal" podStartSLOduration=4.331860455 podStartE2EDuration="4.331860455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:26.331145358 +0000 UTC m=+5.793752230" watchObservedRunningTime="2026-07-24 13:34:26.331860455 +0000 UTC m=+5.794467299"
Jul 24 13:34:26.332271 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.332188    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-88.ec2.internal" podStartSLOduration=4.332176399 podStartE2EDuration="4.332176399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:25.244089781 +0000 UTC m=+4.706696642" watchObservedRunningTime="2026-07-24 13:34:26.332176399 +0000 UTC m=+5.794783244"
Jul 24 13:34:26.568608 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.568521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:26.568786 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.568696    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:26.568786 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.568760    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:30.568741588 +0000 UTC m=+10.031348411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:26.669792 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.669616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:26.669792 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:26.669682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:26.670031 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.669846    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:26.670031 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.669868    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:26.670031 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.669880    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:26.670031 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.669948    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:30.669919714 +0000 UTC m=+10.132526551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:26.670630 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.670393    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:26.670630 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:26.670446    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:30.670430529 +0000 UTC m=+10.133037354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:27.119746 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:27.119713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:27.119911 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:27.119853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:27.120565 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:27.120544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:27.120660 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:27.120644    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:27.120900 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:27.120726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:27.120900 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:27.120801    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:29.118938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:29.118904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:29.119302 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:29.119054    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:29.119465 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:29.119447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:29.119588 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:29.119565    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:29.119720 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:29.119705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:29.119808 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:29.119787    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:30.603321 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:30.603280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:30.603771 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.603452    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:30.603771 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.603534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:38.603515002 +0000 UTC m=+18.066121827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:30.703869 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:30.703831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:30.704048 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:30.703900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:30.704048 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704043    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:30.704163 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704025    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:30.704163 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704062    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:30.704163 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704075    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:30.704163 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:38.704111969 +0000 UTC m=+18.166718793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:30.704163 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:30.704149    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:38.704139692 +0000 UTC m=+18.166746519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:31.119421 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:31.119388    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:31.119643 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:31.119519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:31.119912 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:31.119892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:31.120009 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:31.119989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:31.120088 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:31.120065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:31.120207 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:31.120189    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:33.118840 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:33.118799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:33.119340 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:33.118847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:33.119340 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:33.118810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:33.119340 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:33.118961    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:33.119340 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:33.119015    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:33.119340 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:33.119076    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:35.118743 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:35.118651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:35.119133 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:35.118651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:35.119133 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:35.118796    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:35.119133 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:35.118660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:35.119133 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:35.118929    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:35.119133 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:35.118959    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:37.119005 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:37.118960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:37.119546 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:37.118972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:37.119546 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:37.119119    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:37.119546 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:37.118973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:37.119546 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:37.119204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:37.119546 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:37.119245    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:38.666687 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:38.666579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:38.667079 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.666779    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:38.667079 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.666856    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:34:54.666835033 +0000 UTC m=+34.129441861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:38.767206 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:38.767172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:38.767206 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:38.767215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767338    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767347    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767367    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767379    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767419    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:34:54.76739867 +0000 UTC m=+34.230005492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:38.767449 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:38.767435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:34:54.767428478 +0000 UTC m=+34.230035300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:39.118611 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:39.118523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:39.118770 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:39.118525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:39.118770 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:39.118680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:39.118770 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:39.118533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:39.118770 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:39.118724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:39.118962 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:39.118837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:41.122004 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:41.121971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:41.122423 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:41.121975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:41.122423 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:41.121985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:41.122423 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:41.122204    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:41.122423 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:41.122274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:41.122423 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:41.122096    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:43.118698 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.118377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:43.119395 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.118394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:43.119395 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.118457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:43.119395 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:43.118802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:43.119395 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:43.118888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:43.119395 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:43.118959    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:43.352461 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.352173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj" event={"ID":"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2","Type":"ContainerStarted","Data":"3dc8815e782b5fcd0994c3582edcc74196d02fcc35499d0a5a2839c0e5aae58f"}
Jul 24 13:34:43.353605 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.353571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dwt4j" event={"ID":"fd7ee460-ed31-4b90-97f6-39b857c5dcf2","Type":"ContainerStarted","Data":"737958c0b7be531f87ca50c6ce0159989463c3e78bda4565c9ba4ba8006d334e"}
Jul 24 13:34:43.354982 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.354954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lp8w5" event={"ID":"fee69372-6d61-41d8-bed4-b45a4200af87","Type":"ContainerStarted","Data":"32322644596a86f0f1fbe04b87a5d1673b18545f4682c8467a49969d846a6810"}
Jul 24 13:34:43.356392 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.356359    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="03e98751b3840bf3393e1881a6a61a2cc1afe206bbfffa2043d55c624d96775f" exitCode=0
Jul 24 13:34:43.356526 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.356405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"03e98751b3840bf3393e1881a6a61a2cc1afe206bbfffa2043d55c624d96775f"}
Jul 24 13:34:43.357923 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.357902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h68kb" event={"ID":"f9088c09-c41e-4ee9-aa85-79c17a53b630","Type":"ContainerStarted","Data":"41140741fdad76b894f026cbba1c18428136184cc5c70d554659170a95358f1e"}
Jul 24 13:34:43.359242 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.359216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7qndb" event={"ID":"21c3ce53-0fde-450c-b0f0-28920045b672","Type":"ContainerStarted","Data":"adbbd9aadb87608d6b06f50d618c4dbd935c99697a2d9ed9ed704f2f486e4a62"}
Jul 24 13:34:43.360817 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.360798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s6nth" event={"ID":"d681c4f4-490d-4200-8220-72d44766fa11","Type":"ContainerStarted","Data":"44eb6fa70ced887e6209f7a40e1e31b5e9ff22e30a272f29f0754822ec905ac2"}
Jul 24 13:34:43.362802 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.362787    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:34:43.363159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.363142    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b1f159e-12a1-418a-97d6-8de1de5fa880" containerID="60b5569cb3362d41033054b0d22e8fa483bd884c5cbd8c42b94a8f1708af6dd3" exitCode=1
Jul 24 13:34:43.363209 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.363169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"2283a33e9f417899d5f1cd5f58ca06d016f4433d30081d3022925498b98934fc"}
Jul 24 13:34:43.363244 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.363212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"99901a4eb6cdec7ab9ff8ce45abf8f3adf3c6f1d779b0b618ea85704d48ae12c"}
Jul 24 13:34:43.363244 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.363241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerDied","Data":"60b5569cb3362d41033054b0d22e8fa483bd884c5cbd8c42b94a8f1708af6dd3"}
Jul 24 13:34:43.363306 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.363270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"8de63495c25a6c19be9f6232368f215501a8c745f4f3a4d2256e81b57708c934"}
Jul 24 13:34:43.374149 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.374113    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dwt4j" podStartSLOduration=3.699087257 podStartE2EDuration="22.374100378s" totalImagesPullingTime="18.675013121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:43.373708414 +0000 UTC m=+22.836315259" watchObservedRunningTime="2026-07-24 13:34:43.374100378 +0000 UTC m=+22.836707221"
Jul 24 13:34:43.398016 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.397976    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s6nth" podStartSLOduration=8.164615921 podStartE2EDuration="22.397962846s" totalImagesPullingTime="14.233346925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:43.397733252 +0000 UTC m=+22.860340098" watchObservedRunningTime="2026-07-24 13:34:43.397962846 +0000 UTC m=+22.860569689"
Jul 24 13:34:43.481386 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.481336    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7qndb" podStartSLOduration=3.811254714 podStartE2EDuration="22.481318423s" totalImagesPullingTime="18.670063709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:43.424518981 +0000 UTC m=+22.887125838" watchObservedRunningTime="2026-07-24 13:34:43.481318423 +0000 UTC m=+22.943925296"
Jul 24 13:34:43.521426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.521375    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h68kb" podStartSLOduration=3.833880961 podStartE2EDuration="22.521359004s" totalImagesPullingTime="18.687478043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:43.519914853 +0000 UTC m=+22.982521698" watchObservedRunningTime="2026-07-24 13:34:43.521359004 +0000 UTC m=+22.983965870"
Jul 24 13:34:43.568437 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:43.568387    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lp8w5" podStartSLOduration=3.8976097100000002 podStartE2EDuration="22.568368324s" totalImagesPullingTime="18.670758614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:43.568347143 +0000 UTC m=+23.030953999" watchObservedRunningTime="2026-07-24 13:34:43.568368324 +0000 UTC m=+23.030975167"
Jul 24 13:34:44.370172 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.370038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:34:44.370890 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.370648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"6eeb5ea970b8b61c5979668b8cfa3084495d95849b2d154ff70a257f0e72929c"}
Jul 24 13:34:44.370890 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.370778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"29a8cdeffd0716bac14178fae5bdbfe66b40d772347e52e7191deadfa2476bec"}
Jul 24 13:34:44.373735 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.373702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqjsc" event={"ID":"687056e0-a635-4b5f-b0b2-62588ad14d67","Type":"ContainerStarted","Data":"762e3957e7f104b930929731ca326a151aebd4c33228d2613c503f91d3334d19"}
Jul 24 13:34:44.390931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.390877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqjsc" podStartSLOduration=4.72767318 podStartE2EDuration="23.390858917s" totalImagesPullingTime="18.663185737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:44.390616985 +0000 UTC m=+23.853223830" watchObservedRunningTime="2026-07-24 13:34:44.390858917 +0000 UTC m=+23.853465761"
Jul 24 13:34:44.600521 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.600426    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwt4j_fd7ee460-ed31-4b90-97f6-39b857c5dcf2/dns-node-resolver/0.log"
Jul 24 13:34:44.998158 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:44.998128    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 13:34:45.009893 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.009801    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T13:34:44.998142772Z","UUID":"733a9067-ba01-41a6-9a31-d3815474d044","Handler":null,"Name":"","Endpoint":""}
Jul 24 13:34:45.011411 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.011393    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 13:34:45.011566 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.011434    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 13:34:45.118844 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.118760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:45.119053 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.118761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:45.119053 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:45.118903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:45.119053 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.118772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:45.119053 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:45.119005    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:45.119255 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:45.119104    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:45.362771 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.362742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qndb_21c3ce53-0fde-450c-b0f0-28920045b672/node-ca/0.log"
Jul 24 13:34:45.377388 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:45.377299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj" event={"ID":"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2","Type":"ContainerStarted","Data":"ae3052da79eeb737eb32d3f5b7b5370bf3dca9ce042111ed78429b7b58ba5c30"}
Jul 24 13:34:46.382962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:46.382796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:34:46.383429 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:46.383355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"c5180cd3ddc185e2093c2f7eac2e18593f9110b1804e3c87071a8b4afec2abec"}
Jul 24 13:34:46.427518 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:46.427451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:46.428071 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:46.428045    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:47.118682 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:47.118647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:47.118682 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:47.118670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:47.118969 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:47.118779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:47.118969 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:47.118877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:47.119052 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:47.118965    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:47.119052 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:47.118879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:47.386184 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:47.386156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:47.386727 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:47.386693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s6nth"
Jul 24 13:34:48.390379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.390120    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="37d1dae2a3915761053e78c2a936ed9e2870469afeb4209b1e3706e1ed599d3c" exitCode=0
Jul 24 13:34:48.390379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.390204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"37d1dae2a3915761053e78c2a936ed9e2870469afeb4209b1e3706e1ed599d3c"}
Jul 24 13:34:48.394120 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.394095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:34:48.394528 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.394500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"5a0e3ed462cc19656047b6f8fcd1ea0c39fc14e4d36ab8a141ea962be5c4df61"}
Jul 24 13:34:48.394733 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.394710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:48.394850 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.394777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:48.394959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.394944    2579 scope.go:117] "RemoveContainer" containerID="60b5569cb3362d41033054b0d22e8fa483bd884c5cbd8c42b94a8f1708af6dd3"
Jul 24 13:34:48.398670 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.398629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj" event={"ID":"8db0d2ee-01ba-4838-95de-fc0ee3dff5c2","Type":"ContainerStarted","Data":"010190aae6aea22f9b67e476660c4188f02917b13646adab478eb55c57925f03"}
Jul 24 13:34:48.413012 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.412991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:48.416569 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.416540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:48.507567 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.507516    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvtwj" podStartSLOduration=4.216628838 podStartE2EDuration="27.50750042s" totalImagesPullingTime="23.290871582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:48.507384033 +0000 UTC m=+27.969990882" watchObservedRunningTime="2026-07-24 13:34:48.50750042 +0000 UTC m=+27.970107255"
Jul 24 13:34:48.838118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:48.838079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:34:49.121686 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.121614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:49.121816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.121747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:49.121816 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:49.121751    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:49.121816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.121799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:49.121924 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:49.121876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:49.121984 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:49.121965    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:49.403042 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.402962    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="e616381cf8b5d02cc573cf50eb470d400d24065368ecd6b9bfd4344b0987aed3" exitCode=0
Jul 24 13:34:49.403424 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.403050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"e616381cf8b5d02cc573cf50eb470d400d24065368ecd6b9bfd4344b0987aed3"}
Jul 24 13:34:49.406843 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.406827    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:34:49.407205 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:49.407185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" event={"ID":"0b1f159e-12a1-418a-97d6-8de1de5fa880","Type":"ContainerStarted","Data":"015663ff42c56aa8626a015443d968b107edc01d0538ddafefa17922506ccf03"}
Jul 24 13:34:51.119378 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.119335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:51.119811 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.119449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:51.119811 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:51.119451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:51.119811 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.119469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:51.119811 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:51.119558    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:51.119811 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:51.119619    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:51.414407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.414322    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="1b97173771cca12ad263e78a1eab9ba4a86d145ed7a2495c79f43ad380219da9" exitCode=0
Jul 24 13:34:51.414407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.414399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"1b97173771cca12ad263e78a1eab9ba4a86d145ed7a2495c79f43ad380219da9"}
Jul 24 13:34:51.442073 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:51.442026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq" podStartSLOduration=11.71029847 podStartE2EDuration="30.442009996s" totalImagesPullingTime="18.731711526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:49.503516174 +0000 UTC m=+28.966123017" watchObservedRunningTime="2026-07-24 13:34:51.442009996 +0000 UTC m=+30.904616840"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:53.122513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:53.122610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:53.122626    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:53.122634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:53.122696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:53.122905 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:53.122789    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:54.687094 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:54.687052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:54.687572 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.687186    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:54.687572 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.687252    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret podName:c449e0c2-6d25-4452-b9ba-0e3a08c26fdc nodeName:}" failed. No retries permitted until 2026-07-24 13:35:26.68723503 +0000 UTC m=+66.149841857 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret") pod "global-pull-secret-syncer-j8frr" (UID: "c449e0c2-6d25-4452-b9ba-0e3a08c26fdc") : object "kube-system"/"original-pull-secret" not registered
Jul 24 13:34:54.788331 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:54.788298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:54.788551 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:54.788372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:54.788551 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788471    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:54.788551 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788494    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 13:34:54.788551 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788517    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 13:34:54.788551 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788530    2579 projected.go:196] Error preparing data for projected volume kube-api-access-89lkd for pod openshift-network-diagnostics/network-check-target-djh8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:54.788787 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs podName:a2b1c7ca-a574-466d-b6cf-3a9929c881a1 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:26.788540267 +0000 UTC m=+66.251147110 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs") pod "network-metrics-daemon-77ztq" (UID: "a2b1c7ca-a574-466d-b6cf-3a9929c881a1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 13:34:54.788787 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:54.788588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd podName:a3216655-12ea-492e-a4dd-a893549bebba nodeName:}" failed. No retries permitted until 2026-07-24 13:35:26.788578393 +0000 UTC m=+66.251185217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-89lkd" (UniqueName: "kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd") pod "network-check-target-djh8n" (UID: "a3216655-12ea-492e-a4dd-a893549bebba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 13:34:55.119359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:55.119274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:55.119359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:55.119299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:55.119665 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:55.119274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:55.119665 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:55.119425    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:55.119665 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:55.119582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:55.119845 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:55.119668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:56.429548 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:56.429513    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="4375fab04298877ad20b6aa88930114c5b96340ba0dfc7147aad0d099570f458" exitCode=0
Jul 24 13:34:56.429944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:56.429576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"4375fab04298877ad20b6aa88930114c5b96340ba0dfc7147aad0d099570f458"}
Jul 24 13:34:57.118997 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:57.118789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:57.119293 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:57.118818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:57.119293 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:57.119100    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:57.119293 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:57.118866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:57.119458 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:57.119303    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:57.119458 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:57.119394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:34:57.434631 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:57.434545    2579 generic.go:356] "Generic (PLEG): container finished" podID="736a8f69-fccf-46e0-9731-d5447ca9317c" containerID="3979917e204f2bfcddab8db302293b23f956a05cf1034fa8439f517b421d64d5" exitCode=0
Jul 24 13:34:57.434631 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:57.434604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerDied","Data":"3979917e204f2bfcddab8db302293b23f956a05cf1034fa8439f517b421d64d5"}
Jul 24 13:34:58.440309 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:58.440269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" event={"ID":"736a8f69-fccf-46e0-9731-d5447ca9317c","Type":"ContainerStarted","Data":"15549ca31b9e1e21c91e7f69289bfe64afe1ac2f73aae023704492b354ad8a1f"}
Jul 24 13:34:59.119330 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:59.119289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:34:59.119330 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:59.119310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:34:59.119682 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:34:59.119299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:34:59.119682 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:59.119398    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:34:59.119682 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:59.119513    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:34:59.119682 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:34:59.119610    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:01.120052 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:01.120014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:01.120450 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:01.120113    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:01.120450 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:01.120135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:01.120450 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:01.120160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:01.120450 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:01.120225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:01.120450 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:01.120303    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:03.121706 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:03.121672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:03.121706 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:03.121711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:03.122145 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:03.121720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:03.122145 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:03.121793    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:03.122145 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:03.121854    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:03.122145 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:03.121933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:05.119106 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:05.119060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:05.119106 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:05.119085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:05.119632 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:05.119064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:05.119632 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:05.119168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:05.119632 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:05.119233    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:05.119632 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:05.119295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:07.120777 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:07.120742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:07.121204 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:07.120878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:07.121204 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:07.121093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:07.121204 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:07.121168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:07.121395 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:07.121311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:07.121395 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:07.121384    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:08.509549 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.509469    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6f5pk" podStartSLOduration=18.522815049 podStartE2EDuration="47.50944738s" totalImagesPullingTime="28.986632331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:34:58.485438543 +0000 UTC m=+37.948045387" watchObservedRunningTime="2026-07-24 13:35:08.50944738 +0000 UTC m=+47.972054226"
Jul 24 13:35:08.510598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.510086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8frr"]
Jul 24 13:35:08.510598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.510232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:08.510598 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:08.510345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:08.513248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.513164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-77ztq"]
Jul 24 13:35:08.513372 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.513304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:08.515639 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:08.513458    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:08.515639 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.514985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-djh8n"]
Jul 24 13:35:08.515639 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:08.515123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:08.515639 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:08.515230    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:10.118264 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:10.118227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:10.118264 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:10.118266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:10.118678 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:10.118232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:10.118678 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:10.118343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:10.118678 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:10.118462    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:10.118678 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:10.118549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:12.118687 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.118503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:12.119226 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.118511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:12.119226 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:12.118786    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-77ztq" podUID="a2b1c7ca-a574-466d-b6cf-3a9929c881a1"
Jul 24 13:35:12.119226 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.118546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:12.119226 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:12.118832    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j8frr" podUID="c449e0c2-6d25-4452-b9ba-0e3a08c26fdc"
Jul 24 13:35:12.119226 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:12.118892    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djh8n" podUID="a3216655-12ea-492e-a4dd-a893549bebba"
Jul 24 13:35:12.918289 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.918260    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-88.ec2.internal" event="NodeReady"
Jul 24 13:35:12.918541 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.918380    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 13:35:12.997562 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:12.997530    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dhpbl"]
Jul 24 13:35:13.016069 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.016037    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sknvp"]
Jul 24 13:35:13.016196 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.016168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.025313 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.025291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 13:35:13.027891 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.027874    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 13:35:13.031587 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.031571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 13:35:13.034337 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.034316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 13:35:13.034429 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.034363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.037033 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.037015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnnsn\""
Jul 24 13:35:13.037835 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.037817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 13:35:13.037929 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.037913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 13:35:13.038910 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.038897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kzzx2\""
Jul 24 13:35:13.044705 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.044685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhpbl"]
Jul 24 13:35:13.048173 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.048153    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9dg7q"]
Jul 24 13:35:13.064169 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.064138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sknvp"]
Jul 24 13:35:13.064287 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.064259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.075703 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.075669    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 13:35:13.075829 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.075721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 13:35:13.075829 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.075729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 13:35:13.075829 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.075810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h9qpm\""
Jul 24 13:35:13.076552 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.076532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dg7q"]
Jul 24 13:35:13.120752 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-tmp-dir\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4b30834-dcf5-416b-b746-ca66c3463d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-config-volume\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4b30834-dcf5-416b-b746-ca66c3463d32-crio-socket\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-metrics-tls\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prkl\" (UniqueName: \"kubernetes.io/projected/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-kube-api-access-4prkl\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b30834-dcf5-416b-b746-ca66c3463d32-data-volume\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.121118 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.120999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpf79\" (UniqueName: \"kubernetes.io/projected/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-api-access-hpf79\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4b30834-dcf5-416b-b746-ca66c3463d32-crio-socket\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-metrics-tls\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.221640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prkl\" (UniqueName: \"kubernetes.io/projected/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-kube-api-access-4prkl\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b30834-dcf5-416b-b746-ca66c3463d32-data-volume\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpf79\" (UniqueName: \"kubernetes.io/projected/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-api-access-hpf79\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-tmp-dir\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4b30834-dcf5-416b-b746-ca66c3463d32-crio-socket\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4b30834-dcf5-416b-b746-ca66c3463d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-config-volume\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-cert\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.221959 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.221948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffq9r\" (UniqueName: \"kubernetes.io/projected/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-kube-api-access-ffq9r\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.222291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.222049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4b30834-dcf5-416b-b746-ca66c3463d32-data-volume\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.222291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.222140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-tmp-dir\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.222378 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.222365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.222434 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.222417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-config-volume\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.225746 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.225722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4b30834-dcf5-416b-b746-ca66c3463d32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.225848 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.225760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-metrics-tls\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.236215 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.236192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpf79\" (UniqueName: \"kubernetes.io/projected/d4b30834-dcf5-416b-b746-ca66c3463d32-kube-api-access-hpf79\") pod \"insights-runtime-extractor-dhpbl\" (UID: \"d4b30834-dcf5-416b-b746-ca66c3463d32\") " pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.236335 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.236255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prkl\" (UniqueName: \"kubernetes.io/projected/1bdf7870-c76d-4f29-9b35-fe6f925e1ba1-kube-api-access-4prkl\") pod \"dns-default-sknvp\" (UID: \"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1\") " pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.246451 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.246431    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:13.279804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.279773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:13.279952 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.279882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.283411 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.283387    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 13:35:13.284064 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.284048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 13:35:13.284307 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.284291    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fgvmm\""
Jul 24 13:35:13.284363 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.284325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 13:35:13.284591 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.284572    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 13:35:13.286426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.286193    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 13:35:13.286426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.286217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 13:35:13.286426 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.286249    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 13:35:13.322348 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.322306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffq9r\" (UniqueName: \"kubernetes.io/projected/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-kube-api-access-ffq9r\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.322536 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.322394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-cert\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.324875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.324856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhpbl"
Jul 24 13:35:13.325163 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.325148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-cert\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.333319 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.333298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffq9r\" (UniqueName: \"kubernetes.io/projected/7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c-kube-api-access-ffq9r\") pod \"ingress-canary-9dg7q\" (UID: \"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c\") " pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.343284 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.343261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:13.373030 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.373007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9dg7q"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.422846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wsr\" (UniqueName: \"kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.423006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.423085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.423129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.423153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.423944 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.423178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524065 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524176 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524176 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524275 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524275 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.524275 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.524247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wsr\" (UniqueName: \"kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.526064 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.526032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.527637 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.527392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.527637 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.527448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.528997 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.528931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.529398 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.529373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.542674 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.542649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wsr\" (UniqueName: \"kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr\") pod \"console-868ddbccc9-v6k7t\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") " pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.570842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.570812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhpbl"]
Jul 24 13:35:13.574304 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.574283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sknvp"]
Jul 24 13:35:13.575059 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:13.574970    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b30834_dcf5_416b_b746_ca66c3463d32.slice/crio-acdec42266af31d9bd980fde7fee66f07b9b14c16975df4a1814807a6e63033a WatchSource:0}: Error finding container acdec42266af31d9bd980fde7fee66f07b9b14c16975df4a1814807a6e63033a: Status 404 returned error can't find the container with id acdec42266af31d9bd980fde7fee66f07b9b14c16975df4a1814807a6e63033a
Jul 24 13:35:13.576658 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.576635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9dg7q"]
Jul 24 13:35:13.577744 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:13.577716    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdf7870_c76d_4f29_9b35_fe6f925e1ba1.slice/crio-ec0ec332749a14689000bdee2f6b3228d37f0c551a4616e4bc0a7a82d6403fea WatchSource:0}: Error finding container ec0ec332749a14689000bdee2f6b3228d37f0c551a4616e4bc0a7a82d6403fea: Status 404 returned error can't find the container with id ec0ec332749a14689000bdee2f6b3228d37f0c551a4616e4bc0a7a82d6403fea
Jul 24 13:35:13.586761 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:13.586741    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7584b3a9_a5ee_4a4f_8e60_41e3231f9e2c.slice/crio-b9092e4fe09b03772185d244faf2e542840e8879c2a7555968f73185a9e00312 WatchSource:0}: Error finding container b9092e4fe09b03772185d244faf2e542840e8879c2a7555968f73185a9e00312: Status 404 returned error can't find the container with id b9092e4fe09b03772185d244faf2e542840e8879c2a7555968f73185a9e00312
Jul 24 13:35:13.588710 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.588689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:13.735320 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:13.735110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:13.738112 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:13.738087    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51024b49_3084_4f69_981d_1e831bcf5bf8.slice/crio-5fe4a64af701a319b25781b87560897be9e97cfd25639bc21b4e2b0adab60053 WatchSource:0}: Error finding container 5fe4a64af701a319b25781b87560897be9e97cfd25639bc21b4e2b0adab60053: Status 404 returned error can't find the container with id 5fe4a64af701a319b25781b87560897be9e97cfd25639bc21b4e2b0adab60053
Jul 24 13:35:14.118707 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.118674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:14.119157 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.119132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:14.121097 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.119667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:14.132643 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.131876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvb9z\""
Jul 24 13:35:14.132643 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.131921    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:35:14.132643 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.131876    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 13:35:14.132643 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.132302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:35:14.132643 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.132509    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 13:35:14.132920 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.132694    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8g8q\""
Jul 24 13:35:14.487371 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.487281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dg7q" event={"ID":"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c","Type":"ContainerStarted","Data":"b9092e4fe09b03772185d244faf2e542840e8879c2a7555968f73185a9e00312"}
Jul 24 13:35:14.490440 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.490398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868ddbccc9-v6k7t" event={"ID":"51024b49-3084-4f69-981d-1e831bcf5bf8","Type":"ContainerStarted","Data":"5fe4a64af701a319b25781b87560897be9e97cfd25639bc21b4e2b0adab60053"}
Jul 24 13:35:14.492986 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.492928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sknvp" event={"ID":"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1","Type":"ContainerStarted","Data":"ec0ec332749a14689000bdee2f6b3228d37f0c551a4616e4bc0a7a82d6403fea"}
Jul 24 13:35:14.496082 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:14.496021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhpbl" event={"ID":"d4b30834-dcf5-416b-b746-ca66c3463d32","Type":"ContainerStarted","Data":"acdec42266af31d9bd980fde7fee66f07b9b14c16975df4a1814807a6e63033a"}
Jul 24 13:35:15.180580 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.180545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:35:15.182603 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.182583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.194270 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.194145    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 13:35:15.197354 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.197308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.342092 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.341971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg874\" (UniqueName: \"kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg874\" (UniqueName: \"kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.443338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.445124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.444116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.447842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.446141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.447842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.446926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.447842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.447796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.450571 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.450530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.453217 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.453134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.462123 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.462080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg874\" (UniqueName: \"kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874\") pod \"console-865df9799b-5sbnl\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") " pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:15.497724 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:15.497335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:18.435360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.435324    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-htshn"]
Jul 24 13:35:18.451854 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.451821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.452773 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.452750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-htshn"]
Jul 24 13:35:18.458532 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.458511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 13:35:18.458669 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.458602    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-m9zwd\""
Jul 24 13:35:18.458825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.458809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 13:35:18.458825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.458819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 13:35:18.459007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.458990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 13:35:18.459120 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.459099    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 13:35:18.572335 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.572283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.572524 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.572355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0dd117-9029-4910-9fe5-43ba7eae70f1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.572524 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.572420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcnd\" (UniqueName: \"kubernetes.io/projected/df0dd117-9029-4910-9fe5-43ba7eae70f1-kube-api-access-2xcnd\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.572524 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.572469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.674825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.674167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.674825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.674227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0dd117-9029-4910-9fe5-43ba7eae70f1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.674825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.674257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcnd\" (UniqueName: \"kubernetes.io/projected/df0dd117-9029-4910-9fe5-43ba7eae70f1-kube-api-access-2xcnd\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.674825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.674288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.675676 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.675654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0dd117-9029-4910-9fe5-43ba7eae70f1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.677312 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.677262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.677501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.677458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df0dd117-9029-4910-9fe5-43ba7eae70f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.685392 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.685345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcnd\" (UniqueName: \"kubernetes.io/projected/df0dd117-9029-4910-9fe5-43ba7eae70f1-kube-api-access-2xcnd\") pod \"prometheus-operator-84d69f794d-htshn\" (UID: \"df0dd117-9029-4910-9fe5-43ba7eae70f1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:18.763257 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:18.763204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn"
Jul 24 13:35:20.431657 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:20.431629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbhmq"
Jul 24 13:35:20.507730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:20.507695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:35:21.242638 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.241638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-htshn"]
Jul 24 13:35:21.248234 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:21.248200    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0dd117_9029_4910_9fe5_43ba7eae70f1.slice/crio-a69b46aea50092b03e52a778fe7ec4df22c73ab3f3f7ec208d3a0ac89ee7b7cc WatchSource:0}: Error finding container a69b46aea50092b03e52a778fe7ec4df22c73ab3f3f7ec208d3a0ac89ee7b7cc: Status 404 returned error can't find the container with id a69b46aea50092b03e52a778fe7ec4df22c73ab3f3f7ec208d3a0ac89ee7b7cc
Jul 24 13:35:21.523284 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.523244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868ddbccc9-v6k7t" event={"ID":"51024b49-3084-4f69-981d-1e831bcf5bf8","Type":"ContainerStarted","Data":"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"}
Jul 24 13:35:21.524983 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.524961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sknvp" event={"ID":"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1","Type":"ContainerStarted","Data":"599ed7e3db0421459c49770d0537af958f11b1cef9d2887538232e473f10f690"}
Jul 24 13:35:21.525107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.524999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sknvp" event={"ID":"1bdf7870-c76d-4f29-9b35-fe6f925e1ba1","Type":"ContainerStarted","Data":"00f126f6a3c81b6aeaff0c40f5a0a0fcf412fc297e00532e2ece807125413085"}
Jul 24 13:35:21.525107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.525045    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:21.526263 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.526242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhpbl" event={"ID":"d4b30834-dcf5-416b-b746-ca66c3463d32","Type":"ContainerStarted","Data":"3061840cfd6539c44f26edc62fe9f788b8a130132049595147c195cbe91318bb"}
Jul 24 13:35:21.527646 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.527626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865df9799b-5sbnl" event={"ID":"aec47155-4869-49b1-b01b-9bae0eb377ca","Type":"ContainerStarted","Data":"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"}
Jul 24 13:35:21.527720 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.527663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865df9799b-5sbnl" event={"ID":"aec47155-4869-49b1-b01b-9bae0eb377ca","Type":"ContainerStarted","Data":"17bfd05f02056b28b6b83d1f9d62906e21ff6186543ef3836614bb2b57a068e1"}
Jul 24 13:35:21.529017 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.528998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9dg7q" event={"ID":"7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c","Type":"ContainerStarted","Data":"525cdd0bccf54106b3fe2da7c29940af83e987b011cec55357e0651a7759f705"}
Jul 24 13:35:21.529917 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.529898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn" event={"ID":"df0dd117-9029-4910-9fe5-43ba7eae70f1","Type":"ContainerStarted","Data":"a69b46aea50092b03e52a778fe7ec4df22c73ab3f3f7ec208d3a0ac89ee7b7cc"}
Jul 24 13:35:21.545963 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.545912    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868ddbccc9-v6k7t" podStartSLOduration=1.248689283 podStartE2EDuration="8.54589908s" totalImagesPullingTime="7.297209797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:21.545448397 +0000 UTC m=+61.008055244" watchObservedRunningTime="2026-07-24 13:35:21.54589908 +0000 UTC m=+61.008505902"
Jul 24 13:35:21.566437 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.566347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865df9799b-5sbnl" podStartSLOduration=6.56633654 podStartE2EDuration="6.56633654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:21.565862189 +0000 UTC m=+61.028469033" watchObservedRunningTime="2026-07-24 13:35:21.56633654 +0000 UTC m=+61.028943362"
Jul 24 13:35:21.595602 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.595558    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sknvp" podStartSLOduration=2.140586207 podStartE2EDuration="9.595541992s" totalImagesPullingTime="7.454955785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:21.594548887 +0000 UTC m=+61.057155746" watchObservedRunningTime="2026-07-24 13:35:21.595541992 +0000 UTC m=+61.058148833"
Jul 24 13:35:21.622089 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:21.622034    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9dg7q" podStartSLOduration=2.176012908 podStartE2EDuration="9.622018412s" totalImagesPullingTime="7.446005504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:21.621620289 +0000 UTC m=+61.084227153" watchObservedRunningTime="2026-07-24 13:35:21.622018412 +0000 UTC m=+61.084625256"
Jul 24 13:35:23.541826 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.541789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn" event={"ID":"df0dd117-9029-4910-9fe5-43ba7eae70f1","Type":"ContainerStarted","Data":"def6128df5b4c08a833153922b63bb0ee46e30fec583edd9e3d5eb493eb6eb70"}
Jul 24 13:35:23.542291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.541850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn" event={"ID":"df0dd117-9029-4910-9fe5-43ba7eae70f1","Type":"ContainerStarted","Data":"e92535f205ea70f1ad34f1eaf2ffecc8fde6b157a2e910484e53c43c988c21fe"}
Jul 24 13:35:23.566638 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.566569    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-htshn" podStartSLOduration=4.201724361 podStartE2EDuration="5.566544962s" totalImagesPullingTime="1.364820601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:23.565339367 +0000 UTC m=+63.027946229" watchObservedRunningTime="2026-07-24 13:35:23.566544962 +0000 UTC m=+63.029151803"
Jul 24 13:35:23.588971 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.588932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:23.589124 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.589020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:23.590747 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.590717    2579 patch_prober.go:28] interesting pod/console-868ddbccc9-v6k7t container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body=
Jul 24 13:35:23.590867 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:23.590767    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-868ddbccc9-v6k7t" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused"
Jul 24 13:35:25.498621 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.498563    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:25.499134 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.498939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:25.500359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.500329    2579 patch_prober.go:28] interesting pod/console-865df9799b-5sbnl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body=
Jul 24 13:35:25.500520 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.500381    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-865df9799b-5sbnl" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused"
Jul 24 13:35:25.551288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.551250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhpbl" event={"ID":"d4b30834-dcf5-416b-b746-ca66c3463d32","Type":"ContainerStarted","Data":"530b8d3576125952f562d52cf1cf5d992a1e17f9156d6813ef0dc0cde0a6ff25"}
Jul 24 13:35:25.851692 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.851658    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"]
Jul 24 13:35:25.863690 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.863652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:25.866639 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.866610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 24 13:35:25.866789 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.866610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-m66cl\""
Jul 24 13:35:25.866789 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.866611    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 24 13:35:25.870246 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.870219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"]
Jul 24 13:35:25.919349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.919310    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nbfzr"]
Jul 24 13:35:25.939590 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.939556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:25.939905 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.939882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:25.939992 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.939928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:25.940044 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.940004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cbaf6f-ccc2-4330-b925-a09fffbbf140-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:25.940099 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.940044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ql4\" (UniqueName: \"kubernetes.io/projected/18cbaf6f-ccc2-4330-b925-a09fffbbf140-kube-api-access-k6ql4\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:25.942361 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.942337    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 13:35:25.942783 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.942768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 13:35:25.942940 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.942924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 13:35:25.943064 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:25.943045    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mwmx5\""
Jul 24 13:35:26.041014 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.040981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrzr\" (UniqueName: \"kubernetes.io/projected/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-kube-api-access-jnrzr\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ql4\" (UniqueName: \"kubernetes.io/projected/18cbaf6f-ccc2-4330-b925-a09fffbbf140-kube-api-access-k6ql4\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.041211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-metrics-client-ca\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041299    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-root\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.041592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.041592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-sys\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-textfile\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041592 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-wtmp\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.041773 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.041651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cbaf6f-ccc2-4330-b925-a09fffbbf140-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.042291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.042270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cbaf6f-ccc2-4330-b925-a09fffbbf140-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.044326 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.044301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.044470 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.044444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/18cbaf6f-ccc2-4330-b925-a09fffbbf140-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.057721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.057697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ql4\" (UniqueName: \"kubernetes.io/projected/18cbaf6f-ccc2-4330-b925-a09fffbbf140-kube-api-access-k6ql4\") pod \"openshift-state-metrics-b8f49f5c6-rsmdb\" (UID: \"18cbaf6f-ccc2-4330-b925-a09fffbbf140\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.143109 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143109 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143109 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrzr\" (UniqueName: \"kubernetes.io/projected/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-kube-api-access-jnrzr\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-metrics-client-ca\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-root\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-sys\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-textfile\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-wtmp\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143789 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-wtmp\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143789 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-root\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.143789 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.143637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-sys\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.144043 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.144014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-metrics-client-ca\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.144153 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:26.144137    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 24 13:35:26.144228 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:26.144216    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls podName:9f938bc7-c7ce-4d4d-a383-2c2d7444fe09 nodeName:}" failed. No retries permitted until 2026-07-24 13:35:26.644197095 +0000 UTC m=+66.106803932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls") pod "node-exporter-nbfzr" (UID: "9f938bc7-c7ce-4d4d-a383-2c2d7444fe09") : secret "node-exporter-tls" not found
Jul 24 13:35:26.144294 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.144135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-textfile\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.144515 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.144493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.145774 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.145747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.163196 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.163162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrzr\" (UniqueName: \"kubernetes.io/projected/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-kube-api-access-jnrzr\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.174913 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.174883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"
Jul 24 13:35:26.386926 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.386884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb"]
Jul 24 13:35:26.390139 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:26.390108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cbaf6f_ccc2_4330_b925_a09fffbbf140.slice/crio-1c10732b39fc33af67c2e7af76eb3b2972e55c588db728fe4d8d3af0f5a32138 WatchSource:0}: Error finding container 1c10732b39fc33af67c2e7af76eb3b2972e55c588db728fe4d8d3af0f5a32138: Status 404 returned error can't find the container with id 1c10732b39fc33af67c2e7af76eb3b2972e55c588db728fe4d8d3af0f5a32138
Jul 24 13:35:26.557634 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.557603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb" event={"ID":"18cbaf6f-ccc2-4330-b925-a09fffbbf140","Type":"ContainerStarted","Data":"23020992400f23bb97b33de6051e4985ba5a29b3394eb89a2b3e0b1d580d486e"}
Jul 24 13:35:26.558078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.557679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb" event={"ID":"18cbaf6f-ccc2-4330-b925-a09fffbbf140","Type":"ContainerStarted","Data":"1c10732b39fc33af67c2e7af76eb3b2972e55c588db728fe4d8d3af0f5a32138"}
Jul 24 13:35:26.647613 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.647578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.650133 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.650100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9f938bc7-c7ce-4d4d-a383-2c2d7444fe09-node-exporter-tls\") pod \"node-exporter-nbfzr\" (UID: \"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09\") " pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.748370 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.748278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:26.751095 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.751062    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 13:35:26.761856 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.761828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c449e0c2-6d25-4452-b9ba-0e3a08c26fdc-original-pull-secret\") pod \"global-pull-secret-syncer-j8frr\" (UID: \"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc\") " pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:26.849045 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.849002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:26.849238 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.849063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:26.849518 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.849492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbfzr"
Jul 24 13:35:26.852546 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.852523    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 13:35:26.852723 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.852595    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 13:35:26.861832 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.861791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b1c7ca-a574-466d-b6cf-3a9929c881a1-metrics-certs\") pod \"network-metrics-daemon-77ztq\" (UID: \"a2b1c7ca-a574-466d-b6cf-3a9929c881a1\") " pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:26.865277 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.865250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 13:35:26.872436 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:26.872416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lkd\" (UniqueName: \"kubernetes.io/projected/a3216655-12ea-492e-a4dd-a893549bebba-kube-api-access-89lkd\") pod \"network-check-target-djh8n\" (UID: \"a3216655-12ea-492e-a4dd-a893549bebba\") " pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:27.040706 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.040653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8g8q\""
Jul 24 13:35:27.048787 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.048747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-77ztq"
Jul 24 13:35:27.052768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.052736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvb9z\""
Jul 24 13:35:27.057791 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.057573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j8frr"
Jul 24 13:35:27.059271 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.059238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:27.563264 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:27.563224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb" event={"ID":"18cbaf6f-ccc2-4330-b925-a09fffbbf140","Type":"ContainerStarted","Data":"a28d7bef5b2f4cd064141969c6575a0e3b4cc52a9f28294d29c2707010f252c9"}
Jul 24 13:35:28.503172 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:28.503144    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f938bc7_c7ce_4d4d_a383_2c2d7444fe09.slice/crio-eaab8689b3bd0860e440480ea71a71b20cdda91c064c11b1479ded8e014a3b6e WatchSource:0}: Error finding container eaab8689b3bd0860e440480ea71a71b20cdda91c064c11b1479ded8e014a3b6e: Status 404 returned error can't find the container with id eaab8689b3bd0860e440480ea71a71b20cdda91c064c11b1479ded8e014a3b6e
Jul 24 13:35:28.583075 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.583008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbfzr" event={"ID":"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09","Type":"ContainerStarted","Data":"eaab8689b3bd0860e440480ea71a71b20cdda91c064c11b1479ded8e014a3b6e"}
Jul 24 13:35:28.657792 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.657754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-djh8n"]
Jul 24 13:35:28.663794 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:28.663755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3216655_12ea_492e_a4dd_a893549bebba.slice/crio-e404b9c45bc0add6b380b30f1c721c6ce5466dc134456fff5e7faa801f78dd1d WatchSource:0}: Error finding container e404b9c45bc0add6b380b30f1c721c6ce5466dc134456fff5e7faa801f78dd1d: Status 404 returned error can't find the container with id e404b9c45bc0add6b380b30f1c721c6ce5466dc134456fff5e7faa801f78dd1d
Jul 24 13:35:28.690868 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.690831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-77ztq"]
Jul 24 13:35:28.694722 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:28.694691    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b1c7ca_a574_466d_b6cf_3a9929c881a1.slice/crio-e5433e65f3732e39085ddab0badd76f91bbddb46ee45b99b54549f59990cf332 WatchSource:0}: Error finding container e5433e65f3732e39085ddab0badd76f91bbddb46ee45b99b54549f59990cf332: Status 404 returned error can't find the container with id e5433e65f3732e39085ddab0badd76f91bbddb46ee45b99b54549f59990cf332
Jul 24 13:35:28.697233 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.697205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j8frr"]
Jul 24 13:35:28.701418 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:28.701390    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc449e0c2_6d25_4452_b9ba_0e3a08c26fdc.slice/crio-5bfe33e217023035860b3b740a11a95b739115b71f126cc486a77b860283af12 WatchSource:0}: Error finding container 5bfe33e217023035860b3b740a11a95b739115b71f126cc486a77b860283af12: Status 404 returned error can't find the container with id 5bfe33e217023035860b3b740a11a95b739115b71f126cc486a77b860283af12
Jul 24 13:35:28.964166 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.964127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"]
Jul 24 13:35:28.967061 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.967032    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.970988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.971272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.971658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b4fnh\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.971842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.972041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bn0s8f9c6k9ms\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.972210    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 13:35:28.972904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.972391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 13:35:28.981828 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:28.981803    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"]
Jul 24 13:35:29.068888 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.068846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.068888 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.068893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069139 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.068962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069139 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.069028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkrpl\" (UniqueName: \"kubernetes.io/projected/9719e502-889e-41ac-8721-fe00fcac7d7e-kube-api-access-nkrpl\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069139 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.069057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069139 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.069085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-grpc-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069139 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.069118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9719e502-889e-41ac-8721-fe00fcac7d7e-metrics-client-ca\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.069359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.069144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkrpl\" (UniqueName: \"kubernetes.io/projected/9719e502-889e-41ac-8721-fe00fcac7d7e-kube-api-access-nkrpl\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-grpc-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9719e502-889e-41ac-8721-fe00fcac7d7e-metrics-client-ca\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.170831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.171913 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.171887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9719e502-889e-41ac-8721-fe00fcac7d7e-metrics-client-ca\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.177042 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.176523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.177042 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.176642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.177042 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.176724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.177042 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.177000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.177332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.177095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.178151 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.178097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9719e502-889e-41ac-8721-fe00fcac7d7e-secret-grpc-tls\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.185953 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.185919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkrpl\" (UniqueName: \"kubernetes.io/projected/9719e502-889e-41ac-8721-fe00fcac7d7e-kube-api-access-nkrpl\") pod \"thanos-querier-6f7bb66f4d-mh8rf\" (UID: \"9719e502-889e-41ac-8721-fe00fcac7d7e\") " pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.279617 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.279577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:29.551829 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.551801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"]
Jul 24 13:35:29.590793 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.590708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77ztq" event={"ID":"a2b1c7ca-a574-466d-b6cf-3a9929c881a1","Type":"ContainerStarted","Data":"e5433e65f3732e39085ddab0badd76f91bbddb46ee45b99b54549f59990cf332"}
Jul 24 13:35:29.592816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.592782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8frr" event={"ID":"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc","Type":"ContainerStarted","Data":"5bfe33e217023035860b3b740a11a95b739115b71f126cc486a77b860283af12"}
Jul 24 13:35:29.595719 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.595691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhpbl" event={"ID":"d4b30834-dcf5-416b-b746-ca66c3463d32","Type":"ContainerStarted","Data":"0fe3533caa2bc2532dcd36917a56bc1ff4f24f33b5c7005dcc4b69393cd3cf3b"}
Jul 24 13:35:29.597345 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.597314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-djh8n" event={"ID":"a3216655-12ea-492e-a4dd-a893549bebba","Type":"ContainerStarted","Data":"e404b9c45bc0add6b380b30f1c721c6ce5466dc134456fff5e7faa801f78dd1d"}
Jul 24 13:35:29.601066 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.601037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb" event={"ID":"18cbaf6f-ccc2-4330-b925-a09fffbbf140","Type":"ContainerStarted","Data":"0ddc2270fb59b7695c729a4e5b5fce041735560da5af7aa8f12d491eabb51d1b"}
Jul 24 13:35:29.644795 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.644731    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dhpbl" podStartSLOduration=2.869372304 podStartE2EDuration="17.644704747s" totalImagesPullingTime="14.775332443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:35:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:29.62017764 +0000 UTC m=+69.082784488" watchObservedRunningTime="2026-07-24 13:35:29.644704747 +0000 UTC m=+69.107311595"
Jul 24 13:35:29.645368 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:29.645328    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rsmdb" podStartSLOduration=2.179569109 podStartE2EDuration="4.645307884s" totalImagesPullingTime="2.465738775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:29.642976516 +0000 UTC m=+69.105583361" watchObservedRunningTime="2026-07-24 13:35:29.645307884 +0000 UTC m=+69.107914730"
Jul 24 13:35:29.756280 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:29.750899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9719e502_889e_41ac_8721_fe00fcac7d7e.slice/crio-37ce03c70ceca22a22a6653c405123a0a111e638bdfd30bb84cb744d75857749 WatchSource:0}: Error finding container 37ce03c70ceca22a22a6653c405123a0a111e638bdfd30bb84cb744d75857749: Status 404 returned error can't find the container with id 37ce03c70ceca22a22a6653c405123a0a111e638bdfd30bb84cb744d75857749
Jul 24 13:35:30.609549 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.608793    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f938bc7-c7ce-4d4d-a383-2c2d7444fe09" containerID="800b044d72f7d344940e1c69068ad0953397e56f440c1bcde39b4c32b6f68737" exitCode=0
Jul 24 13:35:30.609549 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.609113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbfzr" event={"ID":"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09","Type":"ContainerDied","Data":"800b044d72f7d344940e1c69068ad0953397e56f440c1bcde39b4c32b6f68737"}
Jul 24 13:35:30.612630 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.612176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77ztq" event={"ID":"a2b1c7ca-a574-466d-b6cf-3a9929c881a1","Type":"ContainerStarted","Data":"cb0e125fdb4e858604f5d0147dfab8619cbdd646d4375193d8339508e73be4ae"}
Jul 24 13:35:30.612630 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.612226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-77ztq" event={"ID":"a2b1c7ca-a574-466d-b6cf-3a9929c881a1","Type":"ContainerStarted","Data":"ebb17b7cf5a41d36211c0ccf8803a06cbc12d2403e0e97b5725364108b2a0975"}
Jul 24 13:35:30.614465 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.614439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"37ce03c70ceca22a22a6653c405123a0a111e638bdfd30bb84cb744d75857749"}
Jul 24 13:35:30.678327 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.677884    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:30.734745 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.734645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-77ztq" podStartSLOduration=68.625860296 podStartE2EDuration="1m9.734623554s" totalImagesPullingTime="1.108763258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:30.732881146 +0000 UTC m=+70.195487991" watchObservedRunningTime="2026-07-24 13:35:30.734623554 +0000 UTC m=+70.197230399"
Jul 24 13:35:30.763307 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.763278    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:35:30.768279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.768012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.783887 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.783861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785704    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-422kq\" (UniqueName: \"kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.786689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.785842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.887545 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.886497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-422kq\" (UniqueName: \"kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.888155 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.887860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.888155 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.887928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.888458 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.888431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.889414 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.889290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.890021 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.889975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.890941 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.890923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:30.897822 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:30.897728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-422kq\" (UniqueName: \"kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq\") pod \"console-67674fd85f-s5zt2\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") " pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:31.080191 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.080146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:31.263229 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.263194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:35:31.268337 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:31.268309    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda673c4ef_6723_4e3c_8514_320347bf80e0.slice/crio-4f4c1120d0f0b5bd804823030dcca623bcd4b64d95c2db4dc70151710af98133 WatchSource:0}: Error finding container 4f4c1120d0f0b5bd804823030dcca623bcd4b64d95c2db4dc70151710af98133: Status 404 returned error can't find the container with id 4f4c1120d0f0b5bd804823030dcca623bcd4b64d95c2db4dc70151710af98133
Jul 24 13:35:31.538159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.538128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sknvp"
Jul 24 13:35:31.621471 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.621383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbfzr" event={"ID":"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09","Type":"ContainerStarted","Data":"910ee541f0523a7dae8917a34e614cb04b022ab87b252ed09908ea7c9beb2f43"}
Jul 24 13:35:31.621816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.621514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbfzr" event={"ID":"9f938bc7-c7ce-4d4d-a383-2c2d7444fe09","Type":"ContainerStarted","Data":"8e0c3ea5e359016e49e2bfea0e48ce949e5ed69e401a7ce004d347fecba99bad"}
Jul 24 13:35:31.625939 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.625906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67674fd85f-s5zt2" event={"ID":"a673c4ef-6723-4e3c-8514-320347bf80e0","Type":"ContainerStarted","Data":"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"}
Jul 24 13:35:31.626084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.625956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67674fd85f-s5zt2" event={"ID":"a673c4ef-6723-4e3c-8514-320347bf80e0","Type":"ContainerStarted","Data":"4f4c1120d0f0b5bd804823030dcca623bcd4b64d95c2db4dc70151710af98133"}
Jul 24 13:35:31.676746 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.676682    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67674fd85f-s5zt2" podStartSLOduration=1.676660446 podStartE2EDuration="1.676660446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:31.675917379 +0000 UTC m=+71.138524225" watchObservedRunningTime="2026-07-24 13:35:31.676660446 +0000 UTC m=+71.139267294"
Jul 24 13:35:31.677239 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:31.677211    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nbfzr" podStartSLOduration=5.331465806 podStartE2EDuration="6.677197633s" totalImagesPullingTime="1.253677062s" totalInitContainerRuntime="92.054765ms" isStatefulPod=true podCreationTimestamp="2026-07-24 13:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:31.649830823 +0000 UTC m=+71.112437685" watchObservedRunningTime="2026-07-24 13:35:31.677197633 +0000 UTC m=+71.139804496"
Jul 24 13:35:32.296622 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.296588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:35:32.300639 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.300604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.307804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.306731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 13:35:32.307804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.307046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 13:35:32.307804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.307276    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 13:35:32.307804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.307443    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 13:35:32.307804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.307653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.309215    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-495p5366k7iol\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.310099    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.310432    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.310675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4nknl\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.310837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311127    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311551    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 13:35:32.311886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.311689    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 13:35:32.313219 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.313201    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 13:35:32.349830 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.349721    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:35:32.398987 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.398953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.398987 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.398986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrkj\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.399561 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.399490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.500149 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrkj\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.500348 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.500898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.501916 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.501580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.502652 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.502624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.503506 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.503348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.504188 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.503861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.504503 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.504394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.505163 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.505133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.505297 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.505265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.505704 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.505677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.507054 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.507031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.507165 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.507082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.507949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.508738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.509899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.510337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.510585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.512407 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.512390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrkj\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.513129 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.513104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.513719 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.513668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.514247 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.514226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.514527 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.514373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.516989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.516947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:32.615431 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:32.615351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:34.934804 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:34.934728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:35:34.935422 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:35:34.935396    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad671153_a97f_4beb_b762_06cc90f9e611.slice/crio-8df640bd12a845811b918b9e324d714547964c01596d7f18b254c33559d6f22b WatchSource:0}: Error finding container 8df640bd12a845811b918b9e324d714547964c01596d7f18b254c33559d6f22b: Status 404 returned error can't find the container with id 8df640bd12a845811b918b9e324d714547964c01596d7f18b254c33559d6f22b
Jul 24 13:35:35.504385 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.504330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:35.510836 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.510802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:35:35.642256 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.642221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-djh8n" event={"ID":"a3216655-12ea-492e-a4dd-a893549bebba","Type":"ContainerStarted","Data":"13620b3001dd193cf7e0062c93dc9295ce45b18d96f49e14f8d60659e8a9e9d9"}
Jul 24 13:35:35.642418 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.642335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:35:35.644522 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.644465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"877c9cac8b96fb23cef7e952108073c533f7e715d364abc7108bbd7a274e151e"}
Jul 24 13:35:35.644667 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.644570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"6e5cb21440a52d8be752dce6cd31a612e76a257dae6da65b9a3563c39c51e337"}
Jul 24 13:35:35.644667 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.644610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"59a4e6becda7311e28b65bda708e410a85c8df85c8c3ba5c52875734cf8fb56c"}
Jul 24 13:35:35.645691 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.645667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"8df640bd12a845811b918b9e324d714547964c01596d7f18b254c33559d6f22b"}
Jul 24 13:35:35.647049 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.647019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j8frr" event={"ID":"c449e0c2-6d25-4452-b9ba-0e3a08c26fdc","Type":"ContainerStarted","Data":"f1e6cea313c31863be9360ade5434a6179587cb8bff3d7c2ae8745e583ec5972"}
Jul 24 13:35:35.664688 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.664628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-djh8n" podStartSLOduration=68.574167666 podStartE2EDuration="1m14.664607954s" totalImagesPullingTime="6.090440288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:35.663733236 +0000 UTC m=+75.126340080" watchObservedRunningTime="2026-07-24 13:35:35.664607954 +0000 UTC m=+75.127214798"
Jul 24 13:35:35.688090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:35.685525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j8frr" podStartSLOduration=68.61390209 podStartE2EDuration="1m14.685505146s" totalImagesPullingTime="6.071603056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:35.683622015 +0000 UTC m=+75.146228885" watchObservedRunningTime="2026-07-24 13:35:35.685505146 +0000 UTC m=+75.148111992"
Jul 24 13:35:36.653835 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.653793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"f81f4a68ee9500b309c36e7ddb4b655055435df3b94f20463aa7b419a473a531"}
Jul 24 13:35:36.654291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.653899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"b4d87b372230eb21403e9d6532b73e2a9acf4c8d4b419c9e6e7e51b78a6a92bd"}
Jul 24 13:35:36.654291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.653928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" event={"ID":"9719e502-889e-41ac-8721-fe00fcac7d7e","Type":"ContainerStarted","Data":"659e793915feadd16f05f854b9877d212d8e046a5b884d501cefcb5b19f9afca"}
Jul 24 13:35:36.654291 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.653956    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:36.655395 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.655374    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="bc1ae9c9fd14ebf75843a1225d114f0114e5111aa4ccba819eb89a4901b92b23" exitCode=0
Jul 24 13:35:36.655497 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.655458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"bc1ae9c9fd14ebf75843a1225d114f0114e5111aa4ccba819eb89a4901b92b23"}
Jul 24 13:35:36.683121 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:36.683053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf" podStartSLOduration=2.734674974 podStartE2EDuration="8.683032502s" totalImagesPullingTime="5.948357528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:36.680810607 +0000 UTC m=+76.143417465" watchObservedRunningTime="2026-07-24 13:35:36.683032502 +0000 UTC m=+76.145639345"
Jul 24 13:35:40.680361 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"cf8b2c2ff879b8412b92f7670a1163875c277ca8aa38048fcfd69a34dc0f8dcd"}
Jul 24 13:35:40.680750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"3dbf1ca46ada1023fd384f223ce7f44a2e60c4a1f6fd1310c8cd075251810752"}
Jul 24 13:35:40.680750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"2dd4878d24e4facb640ca861d5015a8cd69ff321b7553f4fb3dc02950589679f"}
Jul 24 13:35:40.680750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"bc084bb8dd58a675a3f7224d3868835465916775d7e0374157afd98218523a07"}
Jul 24 13:35:40.680750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"bb838f4fb07fc1177a83b51ace201aeb418022a93ed5fce6319d790c44c63ee1"}
Jul 24 13:35:40.680750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.680566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerStarted","Data":"47ba9bb3b636b7492745af1b5d0eb9f2379047fafa84acb68d45d6c1c8b03895"}
Jul 24 13:35:40.717332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:40.717280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.686102997 podStartE2EDuration="8.717262901s" totalImagesPullingTime="4.007946814s" totalInitContainerRuntime="23.21309ms" isStatefulPod=false podCreationTimestamp="2026-07-24 13:35:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:35:40.715071211 +0000 UTC m=+80.177678081" watchObservedRunningTime="2026-07-24 13:35:40.717262901 +0000 UTC m=+80.179869747"
Jul 24 13:35:41.080813 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:41.080666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:41.080813 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:41.080745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:41.085362 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:41.085339    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:41.687887 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:41.687857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:35:41.751040 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:41.751008    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:35:42.615921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:42.615879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:35:42.668366 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:42.668341    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f7bb66f4d-mh8rf"
Jul 24 13:35:55.710936 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:55.710887    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-868ddbccc9-v6k7t" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerName="console" containerID="cri-o://a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7" gracePeriod=15
Jul 24 13:35:56.488282 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.488258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:56.621931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.621831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.621931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.621878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.621931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.621927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.622210 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.621943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.622210 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.622058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.622210 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.622101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7wsr\" (UniqueName: \"kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr\") pod \"51024b49-3084-4f69-981d-1e831bcf5bf8\" (UID: \"51024b49-3084-4f69-981d-1e831bcf5bf8\") "
Jul 24 13:35:56.622363 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.622316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config" (OuterVolumeSpecName: "console-config") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:35:56.622363 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.622322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca" (OuterVolumeSpecName: "service-ca") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:35:56.622363 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.622335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:35:56.624587 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.624559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr" (OuterVolumeSpecName: "kube-api-access-w7wsr") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "kube-api-access-w7wsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:35:56.624700 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.624604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:35:56.624700 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.624647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "51024b49-3084-4f69-981d-1e831bcf5bf8" (UID: "51024b49-3084-4f69-981d-1e831bcf5bf8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:35:56.723159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723110    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wsr\" (UniqueName: \"kubernetes.io/projected/51024b49-3084-4f69-981d-1e831bcf5bf8-kube-api-access-w7wsr\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.723159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723152    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.723159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723162    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.723647 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723170    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.723647 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723182    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51024b49-3084-4f69-981d-1e831bcf5bf8-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.723647 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.723190    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51024b49-3084-4f69-981d-1e831bcf5bf8-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:35:56.743946 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.743910    2579 generic.go:356] "Generic (PLEG): container finished" podID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerID="a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7" exitCode=0
Jul 24 13:35:56.744084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.743958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868ddbccc9-v6k7t"
Jul 24 13:35:56.744084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.743997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868ddbccc9-v6k7t" event={"ID":"51024b49-3084-4f69-981d-1e831bcf5bf8","Type":"ContainerDied","Data":"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"}
Jul 24 13:35:56.744084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.744051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868ddbccc9-v6k7t" event={"ID":"51024b49-3084-4f69-981d-1e831bcf5bf8","Type":"ContainerDied","Data":"5fe4a64af701a319b25781b87560897be9e97cfd25639bc21b4e2b0adab60053"}
Jul 24 13:35:56.744084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.744071    2579 scope.go:117] "RemoveContainer" containerID="a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"
Jul 24 13:35:56.753181 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.753164    2579 scope.go:117] "RemoveContainer" containerID="a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"
Jul 24 13:35:56.753470 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:35:56.753451    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7\": container with ID starting with a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7 not found: ID does not exist" containerID="a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"
Jul 24 13:35:56.753546 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.753498    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7"} err="failed to get container status \"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7\": rpc error: code = NotFound desc = could not find container \"a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7\": container with ID starting with a1f76715e242d9cac1871cf4b0733d1abc98c532d1a4f499cd9924b61c8637c7 not found: ID does not exist"
Jul 24 13:35:56.776072 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.776037    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:56.785119 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:56.785095    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-868ddbccc9-v6k7t"]
Jul 24 13:35:57.124301 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:35:57.124258    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" path="/var/lib/kubelet/pods/51024b49-3084-4f69-981d-1e831bcf5bf8/volumes"
Jul 24 13:36:06.658454 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:06.658423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-djh8n"
Jul 24 13:36:06.775212 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:06.775167    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-865df9799b-5sbnl" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerName="console" containerID="cri-o://2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94" gracePeriod=15
Jul 24 13:36:07.538940 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.538908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:36:07.715020 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.714937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg874\" (UniqueName: \"kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715020 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.714998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715053    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715116    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715501 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config\") pod \"aec47155-4869-49b1-b01b-9bae0eb377ca\" (UID: \"aec47155-4869-49b1-b01b-9bae0eb377ca\") "
Jul 24 13:36:07.715763 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:07.715763 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:07.715763 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.715741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config" (OuterVolumeSpecName: "console-config") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:07.716171 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.716149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:07.717447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.717426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:07.717544 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.717500    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874" (OuterVolumeSpecName: "kube-api-access-qg874") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "kube-api-access-qg874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:36:07.717544 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.717511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aec47155-4869-49b1-b01b-9bae0eb377ca" (UID: "aec47155-4869-49b1-b01b-9bae0eb377ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:07.787184 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.787150    2579 generic.go:356] "Generic (PLEG): container finished" podID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerID="2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94" exitCode=0
Jul 24 13:36:07.787353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.787199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865df9799b-5sbnl"
Jul 24 13:36:07.787353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.787241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865df9799b-5sbnl" event={"ID":"aec47155-4869-49b1-b01b-9bae0eb377ca","Type":"ContainerDied","Data":"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"}
Jul 24 13:36:07.787353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.787295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865df9799b-5sbnl" event={"ID":"aec47155-4869-49b1-b01b-9bae0eb377ca","Type":"ContainerDied","Data":"17bfd05f02056b28b6b83d1f9d62906e21ff6186543ef3836614bb2b57a068e1"}
Jul 24 13:36:07.787353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.787321    2579 scope.go:117] "RemoveContainer" containerID="2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"
Jul 24 13:36:07.795904 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.795888    2579 scope.go:117] "RemoveContainer" containerID="2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"
Jul 24 13:36:07.796186 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:36:07.796165    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94\": container with ID starting with 2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94 not found: ID does not exist" containerID="2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"
Jul 24 13:36:07.796248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.796195    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94"} err="failed to get container status \"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94\": rpc error: code = NotFound desc = could not find container \"2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94\": container with ID starting with 2e5b52e6207ba3732ee2c3a4f6ef36a90ecac63e643685b7ba9c34bdbb40fc94 not found: ID does not exist"
Jul 24 13:36:07.813653 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.813628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:36:07.816591 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816567    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865df9799b-5sbnl"]
Jul 24 13:36:07.816591 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816583    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816602    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aec47155-4869-49b1-b01b-9bae0eb377ca-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816613    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816621    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816630    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816638    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aec47155-4869-49b1-b01b-9bae0eb377ca-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:07.816740 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:07.816646    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg874\" (UniqueName: \"kubernetes.io/projected/aec47155-4869-49b1-b01b-9bae0eb377ca-kube-api-access-qg874\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:09.123848 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:09.123808    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" path="/var/lib/kubelet/pods/aec47155-4869-49b1-b01b-9bae0eb377ca/volumes"
Jul 24 13:36:32.616027 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:32.615980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:32.636180 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:32.636146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:32.898065 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:32.897980    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:50.749554 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:50.750036 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749898    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="prometheus" containerID="cri-o://47ba9bb3b636b7492745af1b5d0eb9f2379047fafa84acb68d45d6c1c8b03895" gracePeriod=600
Jul 24 13:36:50.750036 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749958    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cf8b2c2ff879b8412b92f7670a1163875c277ca8aa38048fcfd69a34dc0f8dcd" gracePeriod=600
Jul 24 13:36:50.750036 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749968    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="config-reloader" containerID="cri-o://bb838f4fb07fc1177a83b51ace201aeb418022a93ed5fce6319d790c44c63ee1" gracePeriod=600
Jul 24 13:36:50.750036 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.750006    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy" containerID="cri-o://3dbf1ca46ada1023fd384f223ce7f44a2e60c4a1f6fd1310c8cd075251810752" gracePeriod=600
Jul 24 13:36:50.750297 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749938    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-web" containerID="cri-o://2dd4878d24e4facb640ca861d5015a8cd69ff321b7553f4fb3dc02950589679f" gracePeriod=600
Jul 24 13:36:50.750297 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.749952    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="thanos-sidecar" containerID="cri-o://bc084bb8dd58a675a3f7224d3868835465916775d7e0374157afd98218523a07" gracePeriod=600
Jul 24 13:36:50.952806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952773    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="cf8b2c2ff879b8412b92f7670a1163875c277ca8aa38048fcfd69a34dc0f8dcd" exitCode=0
Jul 24 13:36:50.952806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952805    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="3dbf1ca46ada1023fd384f223ce7f44a2e60c4a1f6fd1310c8cd075251810752" exitCode=0
Jul 24 13:36:50.952806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952814    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="2dd4878d24e4facb640ca861d5015a8cd69ff321b7553f4fb3dc02950589679f" exitCode=0
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952820    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="bc084bb8dd58a675a3f7224d3868835465916775d7e0374157afd98218523a07" exitCode=0
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952826    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="bb838f4fb07fc1177a83b51ace201aeb418022a93ed5fce6319d790c44c63ee1" exitCode=0
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952833    2579 generic.go:356] "Generic (PLEG): container finished" podID="ad671153-a97f-4beb-b762-06cc90f9e611" containerID="47ba9bb3b636b7492745af1b5d0eb9f2379047fafa84acb68d45d6c1c8b03895" exitCode=0
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"cf8b2c2ff879b8412b92f7670a1163875c277ca8aa38048fcfd69a34dc0f8dcd"}
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"3dbf1ca46ada1023fd384f223ce7f44a2e60c4a1f6fd1310c8cd075251810752"}
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.952997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"2dd4878d24e4facb640ca861d5015a8cd69ff321b7553f4fb3dc02950589679f"}
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.953037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"bc084bb8dd58a675a3f7224d3868835465916775d7e0374157afd98218523a07"}
Jul 24 13:36:50.953096 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.953077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"bb838f4fb07fc1177a83b51ace201aeb418022a93ed5fce6319d790c44c63ee1"}
Jul 24 13:36:50.953374 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.953113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"47ba9bb3b636b7492745af1b5d0eb9f2379047fafa84acb68d45d6c1c8b03895"}
Jul 24 13:36:50.999017 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:50.998985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:51.051534 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051565    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051594    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.051721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgrkj\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051832    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.051993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052059    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052145    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ad671153-a97f-4beb-b762-06cc90f9e611\" (UID: \"ad671153-a97f-4beb-b762-06cc90f9e611\") "
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.052493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.052382    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-metrics-client-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.053422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.053972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.055185    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.055523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.056415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.057756 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.057555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:36:51.059551 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.059385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.059886 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.059847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.061206 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.061158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.062040 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.061972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:36:51.062040 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.062036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out" (OuterVolumeSpecName: "config-out") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:36:51.062175 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.062091    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.062175 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.062106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj" (OuterVolumeSpecName: "kube-api-access-mgrkj") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "kube-api-access-mgrkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:36:51.062598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.062552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.062775 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.062729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.063083 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.063061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config" (OuterVolumeSpecName: "config") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.067062 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.067012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.075993 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.075965    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config" (OuterVolumeSpecName: "web-config") pod "ad671153-a97f-4beb-b762-06cc90f9e611" (UID: "ad671153-a97f-4beb-b762-06cc90f9e611"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:36:51.152874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152840    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-grpc-tls\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.152874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152870    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152885    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152900    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-web-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152913    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgrkj\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-kube-api-access-mgrkj\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152923    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152935    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-kube-rbac-proxy\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152947    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152959    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152971    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad671153-a97f-4beb-b762-06cc90f9e611-tls-assets\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152982    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.152994    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153006    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153017    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-config-out\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153028    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad671153-a97f-4beb-b762-06cc90f9e611-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153041    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153055    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153070    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ad671153-a97f-4beb-b762-06cc90f9e611-secret-metrics-client-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.153078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.153082    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ad671153-a97f-4beb-b762-06cc90f9e611-prometheus-k8s-db\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:36:51.959530 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.959493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ad671153-a97f-4beb-b762-06cc90f9e611","Type":"ContainerDied","Data":"8df640bd12a845811b918b9e324d714547964c01596d7f18b254c33559d6f22b"}
Jul 24 13:36:51.959530 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.959514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:51.960052 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.959570    2579 scope.go:117] "RemoveContainer" containerID="cf8b2c2ff879b8412b92f7670a1163875c277ca8aa38048fcfd69a34dc0f8dcd"
Jul 24 13:36:51.967158 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.967143    2579 scope.go:117] "RemoveContainer" containerID="3dbf1ca46ada1023fd384f223ce7f44a2e60c4a1f6fd1310c8cd075251810752"
Jul 24 13:36:51.975330 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.975182    2579 scope.go:117] "RemoveContainer" containerID="2dd4878d24e4facb640ca861d5015a8cd69ff321b7553f4fb3dc02950589679f"
Jul 24 13:36:51.985387 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.985355    2579 scope.go:117] "RemoveContainer" containerID="bc084bb8dd58a675a3f7224d3868835465916775d7e0374157afd98218523a07"
Jul 24 13:36:51.993779 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.993756    2579 scope.go:117] "RemoveContainer" containerID="bb838f4fb07fc1177a83b51ace201aeb418022a93ed5fce6319d790c44c63ee1"
Jul 24 13:36:51.994187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.994167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:51.997342 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:51.997321    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:52.001344 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.001325    2579 scope.go:117] "RemoveContainer" containerID="47ba9bb3b636b7492745af1b5d0eb9f2379047fafa84acb68d45d6c1c8b03895"
Jul 24 13:36:52.009116 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.009097    2579 scope.go:117] "RemoveContainer" containerID="bc1ae9c9fd14ebf75843a1225d114f0114e5111aa4ccba819eb89a4901b92b23"
Jul 24 13:36:52.035665 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035627    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:52.035933 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035920    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerName="console"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035936    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerName="console"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035944    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="init-config-reloader"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035950    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="init-config-reloader"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035957    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="prometheus"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="prometheus"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035969    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-web"
Jul 24 13:36:52.035979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035975    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-web"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035984    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="config-reloader"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035989    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="config-reloader"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.035996    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036001    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerName="console"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036014    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerName="console"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036021    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="thanos-sidecar"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036026    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="thanos-sidecar"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036033    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-thanos"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-thanos"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036080    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="config-reloader"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036087    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-web"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036093    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec47155-4869-49b1-b01b-9bae0eb377ca" containerName="console"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036098    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="thanos-sidecar"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036104    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="prometheus"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036114    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" containerName="kube-rbac-proxy-thanos"
Jul 24 13:36:52.036177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.036120    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="51024b49-3084-4f69-981d-1e831bcf5bf8" containerName="console"
Jul 24 13:36:52.039983 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.039966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.047353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 13:36:52.047353 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 13:36:52.047555 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047369    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 13:36:52.047555 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 13:36:52.047555 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047435    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4nknl\""
Jul 24 13:36:52.047698 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 13:36:52.047753 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 13:36:52.047806 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047784    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 13:36:52.047852 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 13:36:52.047911 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.047894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 13:36:52.048894 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.048874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 13:36:52.049029 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.049010    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 13:36:52.053923 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.053904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 13:36:52.054011 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.053914    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 13:36:52.054076 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.054055    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 13:36:52.054369 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.054353    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-495p5366k7iol\""
Jul 24 13:36:52.054957 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.054934    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 13:36:52.074946 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.074917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:52.160406 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160599 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160599 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160599 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-config-out\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160599 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxqf\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-kube-api-access-cjxqf\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.160877 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.161074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.161074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.161074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.161074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.160941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-web-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.261765 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.261765 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-web-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.261973 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.261973 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.261973 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262114 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.261976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-config-out\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262114 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262114 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxqf\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-kube-api-access-cjxqf\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262114 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262114 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262349 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262767 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262767 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.262866 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.262798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.263153 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.263133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.263412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.263385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.263833 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.263813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.264004 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.263979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265084 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-web-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265181 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-config\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265457 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265551 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.265842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.265820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25edc33f-5339-4437-8ffb-cb6b02e8def7-config-out\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.266085 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.266033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.266224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.266199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.266685 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.266661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.267779 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.267754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.268063 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.268042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.268348 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.268327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.268348 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.268343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/25edc33f-5339-4437-8ffb-cb6b02e8def7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.268608 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.268592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/25edc33f-5339-4437-8ffb-cb6b02e8def7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.282148 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.282118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxqf\" (UniqueName: \"kubernetes.io/projected/25edc33f-5339-4437-8ffb-cb6b02e8def7-kube-api-access-cjxqf\") pod \"prometheus-k8s-0\" (UID: \"25edc33f-5339-4437-8ffb-cb6b02e8def7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.349360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.349312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:36:52.510565 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.510531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 13:36:52.514733 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:36:52.514703    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25edc33f_5339_4437_8ffb_cb6b02e8def7.slice/crio-21aeabe9e85d3c7103fef61dd19105675926d40021c032d45195c2371769b2d3 WatchSource:0}: Error finding container 21aeabe9e85d3c7103fef61dd19105675926d40021c032d45195c2371769b2d3: Status 404 returned error can't find the container with id 21aeabe9e85d3c7103fef61dd19105675926d40021c032d45195c2371769b2d3
Jul 24 13:36:52.966056 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.965971    2579 generic.go:356] "Generic (PLEG): container finished" podID="25edc33f-5339-4437-8ffb-cb6b02e8def7" containerID="43fc0a81acd95e4247e2cb01db764323fc2173813746e38526069f4e2b2702d4" exitCode=0
Jul 24 13:36:52.966056 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.966014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerDied","Data":"43fc0a81acd95e4247e2cb01db764323fc2173813746e38526069f4e2b2702d4"}
Jul 24 13:36:52.966423 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:52.966081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"21aeabe9e85d3c7103fef61dd19105675926d40021c032d45195c2371769b2d3"}
Jul 24 13:36:53.124980 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.124950    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad671153-a97f-4beb-b762-06cc90f9e611" path="/var/lib/kubelet/pods/ad671153-a97f-4beb-b762-06cc90f9e611/volumes"
Jul 24 13:36:53.974187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"59a1595e0aa14779a60c0fe085bfe44246e09d26404058989bab83c5107a00db"}
Jul 24 13:36:53.974589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"14d15ebddb83494478cdbca93113686cd15deaf3b24fd9a78c59b8745f3aea4f"}
Jul 24 13:36:53.974589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974274    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"e9d21206ecfb02755cf6c1e376f5b827711771df814ab1d35db832b3147a538a"}
Jul 24 13:36:53.974589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"c954768a74da170b28a2cdd6c5e796b141543fd163a8e4f0a3da70f49e8251ec"}
Jul 24 13:36:53.974589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"51c26557ac41b3614d2f6c0f6298bbe01b58f211884aec95dd6496c7b1ccad35"}
Jul 24 13:36:53.974589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:53.974349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"25edc33f-5339-4437-8ffb-cb6b02e8def7","Type":"ContainerStarted","Data":"0dca5859b16de3c5eb75bf25cd1dadc88b28a9d5e7e371f8d273a9c5fe245b19"}
Jul 24 13:36:54.009727 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.009676    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.991715511 podStartE2EDuration="2.00965789s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.942379ms" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:54.008195016 +0000 UTC m=+153.470801870" watchObservedRunningTime="2026-07-24 13:36:54.00965789 +0000 UTC m=+153.472264734"
Jul 24 13:36:54.128472 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.128436    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:36:54.131115 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.131093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.146437 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.146408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:36:54.185890 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.185856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186041 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.185898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cts\" (UniqueName: \"kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186041 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.185935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186041 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.185966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186041 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.186007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186167 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.186148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.186214 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.186178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287226 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cts\" (UniqueName: \"kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.287416 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.287398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.288137 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.288107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.288261 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.288216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.288330 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.288309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.288371 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.288340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.290057 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.290030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.290121 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.290103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.296535 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.296512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cts\" (UniqueName: \"kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts\") pod \"console-7cf65cc4bf-nnhgv\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") " pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.440438 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.440401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:36:54.573366 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.573339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:36:54.575799 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:36:54.575770    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ee74da_8fac_4f95_b1be_ff173c7d874c.slice/crio-a5058b0ec40f6c658840347fa7227a049fd335897fb6776dec71ae87caf46a9e WatchSource:0}: Error finding container a5058b0ec40f6c658840347fa7227a049fd335897fb6776dec71ae87caf46a9e: Status 404 returned error can't find the container with id a5058b0ec40f6c658840347fa7227a049fd335897fb6776dec71ae87caf46a9e
Jul 24 13:36:54.980447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.980356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf65cc4bf-nnhgv" event={"ID":"c0ee74da-8fac-4f95-b1be-ff173c7d874c","Type":"ContainerStarted","Data":"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"}
Jul 24 13:36:54.980447 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:54.980419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf65cc4bf-nnhgv" event={"ID":"c0ee74da-8fac-4f95-b1be-ff173c7d874c","Type":"ContainerStarted","Data":"a5058b0ec40f6c658840347fa7227a049fd335897fb6776dec71ae87caf46a9e"}
Jul 24 13:36:55.003935 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:55.003882    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cf65cc4bf-nnhgv" podStartSLOduration=1.003865592 podStartE2EDuration="1.003865592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:36:55.002627356 +0000 UTC m=+154.465234200" watchObservedRunningTime="2026-07-24 13:36:55.003865592 +0000 UTC m=+154.466472503"
Jul 24 13:36:57.349654 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:36:57.349615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:37:04.441171 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:04.441121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:37:04.441713 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:04.441192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:37:04.445829 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:04.445808    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:37:05.020400 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:05.020370    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:37:05.086432 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:05.086394    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:37:30.106354 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.106295    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67674fd85f-s5zt2" podUID="a673c4ef-6723-4e3c-8514-320347bf80e0" containerName="console" containerID="cri-o://715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc" gracePeriod=15
Jul 24 13:37:30.354512 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.354458    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:37:30.399610 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399610 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399567    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399610 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-422kq\" (UniqueName: \"kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399662    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399735    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.399874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config\") pod \"a673c4ef-6723-4e3c-8514-320347bf80e0\" (UID: \"a673c4ef-6723-4e3c-8514-320347bf80e0\") "
Jul 24 13:37:30.400046 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.399980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:30.400046 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.400009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:30.400153 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.400099    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.400153 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.400118    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.400339 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.400311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config" (OuterVolumeSpecName: "console-config") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:30.400459 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.400336    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:37:30.402605 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.402578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:37:30.402696 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.402675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq" (OuterVolumeSpecName: "kube-api-access-422kq") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "kube-api-access-422kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:37:30.402696 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.402677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a673c4ef-6723-4e3c-8514-320347bf80e0" (UID: "a673c4ef-6723-4e3c-8514-320347bf80e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:37:30.500875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.500832    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.500875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.500870    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.500875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.500881    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a673c4ef-6723-4e3c-8514-320347bf80e0-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.500875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.500891    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a673c4ef-6723-4e3c-8514-320347bf80e0-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:30.501130 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:30.500899    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-422kq\" (UniqueName: \"kubernetes.io/projected/a673c4ef-6723-4e3c-8514-320347bf80e0-kube-api-access-422kq\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:37:31.111412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.111377    2579 generic.go:356] "Generic (PLEG): container finished" podID="a673c4ef-6723-4e3c-8514-320347bf80e0" containerID="715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc" exitCode=0
Jul 24 13:37:31.111814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.111433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67674fd85f-s5zt2"
Jul 24 13:37:31.111814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.111442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67674fd85f-s5zt2" event={"ID":"a673c4ef-6723-4e3c-8514-320347bf80e0","Type":"ContainerDied","Data":"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"}
Jul 24 13:37:31.111814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.111528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67674fd85f-s5zt2" event={"ID":"a673c4ef-6723-4e3c-8514-320347bf80e0","Type":"ContainerDied","Data":"4f4c1120d0f0b5bd804823030dcca623bcd4b64d95c2db4dc70151710af98133"}
Jul 24 13:37:31.111814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.111549    2579 scope.go:117] "RemoveContainer" containerID="715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"
Jul 24 13:37:31.119812 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.119794    2579 scope.go:117] "RemoveContainer" containerID="715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"
Jul 24 13:37:31.120109 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:37:31.120085    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc\": container with ID starting with 715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc not found: ID does not exist" containerID="715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"
Jul 24 13:37:31.120193 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.120116    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc"} err="failed to get container status \"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc\": rpc error: code = NotFound desc = could not find container \"715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc\": container with ID starting with 715d42980f410538d2c654a1550a3ca0b9a82e4acdd029215556c5a43eb9e1cc not found: ID does not exist"
Jul 24 13:37:31.134067 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.134038    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:37:31.138041 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:31.138020    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67674fd85f-s5zt2"]
Jul 24 13:37:33.122466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:33.122427    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a673c4ef-6723-4e3c-8514-320347bf80e0" path="/var/lib/kubelet/pods/a673c4ef-6723-4e3c-8514-320347bf80e0/volumes"
Jul 24 13:37:52.349943 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:52.349879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:37:52.365671 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:52.365645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:37:53.207248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:37:53.207222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 13:38:04.385059 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.385024    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:38:04.385472 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.385300    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a673c4ef-6723-4e3c-8514-320347bf80e0" containerName="console"
Jul 24 13:38:04.385472 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.385315    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a673c4ef-6723-4e3c-8514-320347bf80e0" containerName="console"
Jul 24 13:38:04.385472 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.385365    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a673c4ef-6723-4e3c-8514-320347bf80e0" containerName="console"
Jul 24 13:38:04.388305 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.388285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.406064 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.406037    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:38:04.493055 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfbw\" (UniqueName: \"kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493055 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.493281 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.493249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594241 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfbw\" (UniqueName: \"kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594241 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594473 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594473 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594622 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594622 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.594716 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.594633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.595141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.595090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.595141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.595105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.595141 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.595109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.595379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.595359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.597090 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.597059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.597212 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.597189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.603381 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.603360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfbw\" (UniqueName: \"kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw\") pod \"console-847b995479-rt8xp\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") " pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.697321 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.697236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:04.836264 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:04.836233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:38:04.839469 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:38:04.839442    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07e15d9_321f_4930_8fa3_bf2a012285df.slice/crio-4c9d136e4eaa747c7dc85fcee5511668a94b7b20b90f08f20bda07539bb8541d WatchSource:0}: Error finding container 4c9d136e4eaa747c7dc85fcee5511668a94b7b20b90f08f20bda07539bb8541d: Status 404 returned error can't find the container with id 4c9d136e4eaa747c7dc85fcee5511668a94b7b20b90f08f20bda07539bb8541d
Jul 24 13:38:05.236971 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:05.236926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b995479-rt8xp" event={"ID":"b07e15d9-321f-4930-8fa3-bf2a012285df","Type":"ContainerStarted","Data":"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"}
Jul 24 13:38:05.237193 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:05.236986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b995479-rt8xp" event={"ID":"b07e15d9-321f-4930-8fa3-bf2a012285df","Type":"ContainerStarted","Data":"4c9d136e4eaa747c7dc85fcee5511668a94b7b20b90f08f20bda07539bb8541d"}
Jul 24 13:38:14.697874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:14.697810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:14.698381 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:14.697893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:14.702781 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:14.702756    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:14.725181 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:14.725119    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847b995479-rt8xp" podStartSLOduration=10.725102291 podStartE2EDuration="10.725102291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:05.262217549 +0000 UTC m=+224.724824393" watchObservedRunningTime="2026-07-24 13:38:14.725102291 +0000 UTC m=+234.187709135"
Jul 24 13:38:15.275979 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:15.275948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:38:15.335175 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:15.335137    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:38:40.360659 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.360611    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cf65cc4bf-nnhgv" podUID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" containerName="console" containerID="cri-o://a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd" gracePeriod=15
Jul 24 13:38:40.592657 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.592633    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:38:40.681044 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.680949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681189 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681189 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681189 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681189 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681159    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7cts\" (UniqueName: \"kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681189 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681427 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle\") pod \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\" (UID: \"c0ee74da-8fac-4f95-b1be-ff173c7d874c\") "
Jul 24 13:38:40.681427 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config" (OuterVolumeSpecName: "console-config") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:40.681665 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681585    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:40.681792 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:40.681792 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.681771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:38:40.683540 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.683510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:38:40.683857 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.683832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:38:40.683937 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.683852    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts" (OuterVolumeSpecName: "kube-api-access-x7cts") pod "c0ee74da-8fac-4f95-b1be-ff173c7d874c" (UID: "c0ee74da-8fac-4f95-b1be-ff173c7d874c"). InnerVolumeSpecName "kube-api-access-x7cts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:38:40.781775 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781735    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cts\" (UniqueName: \"kubernetes.io/projected/c0ee74da-8fac-4f95-b1be-ff173c7d874c-kube-api-access-x7cts\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.781775 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781766    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.781775 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781779    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.782031 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781792    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.782031 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781804    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.782031 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781816    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0ee74da-8fac-4f95-b1be-ff173c7d874c-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:40.782031 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:40.781827    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ee74da-8fac-4f95-b1be-ff173c7d874c-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:38:41.374190 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.374153    2579 generic.go:356] "Generic (PLEG): container finished" podID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" containerID="a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd" exitCode=0
Jul 24 13:38:41.374686 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.374203    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cf65cc4bf-nnhgv"
Jul 24 13:38:41.374686 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.374218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf65cc4bf-nnhgv" event={"ID":"c0ee74da-8fac-4f95-b1be-ff173c7d874c","Type":"ContainerDied","Data":"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"}
Jul 24 13:38:41.374686 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.374258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cf65cc4bf-nnhgv" event={"ID":"c0ee74da-8fac-4f95-b1be-ff173c7d874c","Type":"ContainerDied","Data":"a5058b0ec40f6c658840347fa7227a049fd335897fb6776dec71ae87caf46a9e"}
Jul 24 13:38:41.374686 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.374279    2579 scope.go:117] "RemoveContainer" containerID="a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"
Jul 24 13:38:41.382663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.382643    2579 scope.go:117] "RemoveContainer" containerID="a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"
Jul 24 13:38:41.382973 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:38:41.382952    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd\": container with ID starting with a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd not found: ID does not exist" containerID="a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"
Jul 24 13:38:41.383023 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.382983    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd"} err="failed to get container status \"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd\": rpc error: code = NotFound desc = could not find container \"a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd\": container with ID starting with a5015fc707cfdc38a509071efaf6373856e998f42898534053cfc1b7b1c2d5cd not found: ID does not exist"
Jul 24 13:38:41.392952 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.392923    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:38:41.396563 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:41.396538    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cf65cc4bf-nnhgv"]
Jul 24 13:38:43.123019 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:38:43.122984    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" path="/var/lib/kubelet/pods/c0ee74da-8fac-4f95-b1be-ff173c7d874c/volumes"
Jul 24 13:39:20.948036 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:20.948007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:39:20.948491 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:20.948121    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:39:20.951805 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:20.951778    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 13:39:40.478062 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.478030    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9vkbc"]
Jul 24 13:39:40.480227 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.478307    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" containerName="console"
Jul 24 13:39:40.480227 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.478317    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" containerName="console"
Jul 24 13:39:40.480227 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.478362    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ee74da-8fac-4f95-b1be-ff173c7d874c" containerName="console"
Jul 24 13:39:40.481075 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.481059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.483872 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.483852    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 13:39:40.483872 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.483861    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5cwfl\""
Jul 24 13:39:40.485400 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.485384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 13:39:40.492813 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.492790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9vkbc"]
Jul 24 13:39:40.528928 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.528892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q2v\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-kube-api-access-q2q2v\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.529122 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.528947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-bound-sa-token\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.629949 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.629914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-bound-sa-token\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.630131 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.629987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q2v\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-kube-api-access-q2q2v\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.639078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.639046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-bound-sa-token\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.639230 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.639171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q2v\" (UniqueName: \"kubernetes.io/projected/f0fea21d-07a7-492a-a1a7-cf33015aeeaa-kube-api-access-q2q2v\") pod \"cert-manager-55b4d954d5-9vkbc\" (UID: \"f0fea21d-07a7-492a-a1a7-cf33015aeeaa\") " pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.804201 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.804167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9vkbc"
Jul 24 13:39:40.934492 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.934440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9vkbc"]
Jul 24 13:39:40.938999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:39:40.938972    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0fea21d_07a7_492a_a1a7_cf33015aeeaa.slice/crio-701b30e2a5672c7548a774d4d63aeccb1da0598423ad9e6a23b51d25d8833f60 WatchSource:0}: Error finding container 701b30e2a5672c7548a774d4d63aeccb1da0598423ad9e6a23b51d25d8833f60: Status 404 returned error can't find the container with id 701b30e2a5672c7548a774d4d63aeccb1da0598423ad9e6a23b51d25d8833f60
Jul 24 13:39:40.941223 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:40.941207    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:39:41.595271 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:41.595230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9vkbc" event={"ID":"f0fea21d-07a7-492a-a1a7-cf33015aeeaa","Type":"ContainerStarted","Data":"701b30e2a5672c7548a774d4d63aeccb1da0598423ad9e6a23b51d25d8833f60"}
Jul 24 13:39:42.644387 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.644339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"]
Jul 24 13:39:42.653371 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.653347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.656034 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.655967    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 24 13:39:42.656578 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.656545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"]
Jul 24 13:39:42.657458 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.657151    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 24 13:39:42.657458 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.657183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-rtk7z\""
Jul 24 13:39:42.747955 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.747914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxllh\" (UniqueName: \"kubernetes.io/projected/96fe3022-0140-4921-8d4d-5afaad8ac8be-kube-api-access-mxllh\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.748146 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.748072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fe3022-0140-4921-8d4d-5afaad8ac8be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.849252 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.849203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxllh\" (UniqueName: \"kubernetes.io/projected/96fe3022-0140-4921-8d4d-5afaad8ac8be-kube-api-access-mxllh\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.849441 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.849331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fe3022-0140-4921-8d4d-5afaad8ac8be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.849783 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.849751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96fe3022-0140-4921-8d4d-5afaad8ac8be-tmp\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.858362 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.858342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxllh\" (UniqueName: \"kubernetes.io/projected/96fe3022-0140-4921-8d4d-5afaad8ac8be-kube-api-access-mxllh\") pod \"openshift-lws-operator-fd8ccff4c-glv9t\" (UID: \"96fe3022-0140-4921-8d4d-5afaad8ac8be\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:42.966962 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:42.966872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"
Jul 24 13:39:44.070695 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:44.070673    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t"]
Jul 24 13:39:44.072584 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:39:44.072548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96fe3022_0140_4921_8d4d_5afaad8ac8be.slice/crio-ae4b99d7154b8b31e311a9572342819e4efd245f6a42d704a76b74062ce5afc0 WatchSource:0}: Error finding container ae4b99d7154b8b31e311a9572342819e4efd245f6a42d704a76b74062ce5afc0: Status 404 returned error can't find the container with id ae4b99d7154b8b31e311a9572342819e4efd245f6a42d704a76b74062ce5afc0
Jul 24 13:39:44.614989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:44.614949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9vkbc" event={"ID":"f0fea21d-07a7-492a-a1a7-cf33015aeeaa","Type":"ContainerStarted","Data":"2d8b094780e57d14af3fd1d8373d93d2d4d9b1fad611e45f70ab5cdf5e7405ec"}
Jul 24 13:39:44.616129 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:44.616109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t" event={"ID":"96fe3022-0140-4921-8d4d-5afaad8ac8be","Type":"ContainerStarted","Data":"ae4b99d7154b8b31e311a9572342819e4efd245f6a42d704a76b74062ce5afc0"}
Jul 24 13:39:44.631961 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:44.631916    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9vkbc" podStartSLOduration=1.5646183439999999 podStartE2EDuration="4.631903931s" totalImagesPullingTime="3.067285587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:44.631555008 +0000 UTC m=+324.094161852" watchObservedRunningTime="2026-07-24 13:39:44.631903931 +0000 UTC m=+324.094510776"
Jul 24 13:39:46.626745 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:46.626706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t" event={"ID":"96fe3022-0140-4921-8d4d-5afaad8ac8be","Type":"ContainerStarted","Data":"e0f66f62a0e5f6cfecbebf4c4f498f17005386be27a6250f099350838f248f26"}
Jul 24 13:39:46.645306 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:39:46.645259    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-glv9t" podStartSLOduration=2.193583642 podStartE2EDuration="4.645243065s" totalImagesPullingTime="2.451659423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:46.644384094 +0000 UTC m=+326.106990935" watchObservedRunningTime="2026-07-24 13:39:46.645243065 +0000 UTC m=+326.107849909"
Jul 24 13:40:03.323634 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.323595    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"]
Jul 24 13:40:03.377224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.377194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"]
Jul 24 13:40:03.377376 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.377320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.380059 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.380031    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 13:40:03.380288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.380271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 13:40:03.380571 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.380552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 13:40:03.380668 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.380636    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 13:40:03.380668 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.380648    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fjml2\""
Jul 24 13:40:03.518834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.518801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.519003 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.518840    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.519003 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.518912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.519003 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.518951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.519109 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.519007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g585g\" (UniqueName: \"kubernetes.io/projected/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-kube-api-access-g585g\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620252 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620252 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620511 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620511 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620511 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g585g\" (UniqueName: \"kubernetes.io/projected/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-kube-api-access-g585g\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.620721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.620701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-manifests\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.622748 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.622725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-tmp\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.622856 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.622803    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.622999 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.622940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-webhook-cert\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.641147 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.641116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g585g\" (UniqueName: \"kubernetes.io/projected/5a067aa3-05ca-4af7-88a5-89e5f04ad91e-kube-api-access-g585g\") pod \"opendatahub-operator-controller-manager-85fcbb566-bvcb6\" (UID: \"5a067aa3-05ca-4af7-88a5-89e5f04ad91e\") " pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.688140 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.688105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:03.852676 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:03.852645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"]
Jul 24 13:40:03.856414 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:40:03.856389    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a067aa3_05ca_4af7_88a5_89e5f04ad91e.slice/crio-2d775eeb47e5a5f303ba5cf891d94938fa7b3cbd3bf462df1e745a6743b9d1d9 WatchSource:0}: Error finding container 2d775eeb47e5a5f303ba5cf891d94938fa7b3cbd3bf462df1e745a6743b9d1d9: Status 404 returned error can't find the container with id 2d775eeb47e5a5f303ba5cf891d94938fa7b3cbd3bf462df1e745a6743b9d1d9
Jul 24 13:40:04.702873 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:04.701828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6" event={"ID":"5a067aa3-05ca-4af7-88a5-89e5f04ad91e","Type":"ContainerStarted","Data":"2d775eeb47e5a5f303ba5cf891d94938fa7b3cbd3bf462df1e745a6743b9d1d9"}
Jul 24 13:40:07.716787 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:07.716750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6" event={"ID":"5a067aa3-05ca-4af7-88a5-89e5f04ad91e","Type":"ContainerStarted","Data":"b3476688d7b73b070d4c0519c0e0ccc288f93b77b903cbed10e5ed760fed68d9"}
Jul 24 13:40:08.722379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:08.722337    2579 generic.go:356] "Generic (PLEG): container finished" podID="5a067aa3-05ca-4af7-88a5-89e5f04ad91e" containerID="b3476688d7b73b070d4c0519c0e0ccc288f93b77b903cbed10e5ed760fed68d9" exitCode=0
Jul 24 13:40:08.722851 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:08.722390    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6" event={"ID":"5a067aa3-05ca-4af7-88a5-89e5f04ad91e","Type":"ContainerDied","Data":"b3476688d7b73b070d4c0519c0e0ccc288f93b77b903cbed10e5ed760fed68d9"}
Jul 24 13:40:09.730181 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:09.730142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6" event={"ID":"5a067aa3-05ca-4af7-88a5-89e5f04ad91e","Type":"ContainerStarted","Data":"080e1c76d080479179011d2cfe1ed1a40d4abe3671db6a0f916f5f621eacd604"}
Jul 24 13:40:09.730688 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:09.730278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:09.754823 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:09.754774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6" podStartSLOduration=2.338231734 podStartE2EDuration="6.754756659s" totalImagesPullingTime="3.414190495s" totalInitContainerRuntime="1.00233443s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:09.753285625 +0000 UTC m=+349.215892472" watchObservedRunningTime="2026-07-24 13:40:09.754756659 +0000 UTC m=+349.217363502"
Jul 24 13:40:20.736402 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:20.736372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85fcbb566-bvcb6"
Jul 24 13:40:23.166820 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.166789    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"]
Jul 24 13:40:23.173851 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.173830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.179285 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.179264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 13:40:23.180503 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.180464    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 13:40:23.180503 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.180466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ljhn7\""
Jul 24 13:40:23.180919 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.180895    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 13:40:23.181021 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.180950    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 13:40:23.188801 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.188772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7105e-55dc-4bf2-a756-22626d6c0e72-tmp\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.188932 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.188825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptpd\" (UniqueName: \"kubernetes.io/projected/feb7105e-55dc-4bf2-a756-22626d6c0e72-kube-api-access-nptpd\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.189262 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.189243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb7105e-55dc-4bf2-a756-22626d6c0e72-tls-certs\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.196045 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.196018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"]
Jul 24 13:40:23.290608 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.290557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7105e-55dc-4bf2-a756-22626d6c0e72-tmp\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.290608 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.290608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptpd\" (UniqueName: \"kubernetes.io/projected/feb7105e-55dc-4bf2-a756-22626d6c0e72-kube-api-access-nptpd\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.290855 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.290645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb7105e-55dc-4bf2-a756-22626d6c0e72-tls-certs\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.293140 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.293113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feb7105e-55dc-4bf2-a756-22626d6c0e72-tmp\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.293298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.293280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feb7105e-55dc-4bf2-a756-22626d6c0e72-tls-certs\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.307814 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.307789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptpd\" (UniqueName: \"kubernetes.io/projected/feb7105e-55dc-4bf2-a756-22626d6c0e72-kube-api-access-nptpd\") pod \"kube-auth-proxy-85dc78bf9-jflxb\" (UID: \"feb7105e-55dc-4bf2-a756-22626d6c0e72\") " pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.486219 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.486135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"
Jul 24 13:40:23.599640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.599604    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"]
Jul 24 13:40:23.601840 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.601809    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.604897 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.604864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 24 13:40:23.604897 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.604879    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zm78x\""
Jul 24 13:40:23.604897 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.604878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 24 13:40:23.605171 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.604964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 24 13:40:23.615922 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.615899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"]
Jul 24 13:40:23.632529 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.632500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb"]
Jul 24 13:40:23.635866 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:40:23.635841    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb7105e_55dc_4bf2_a756_22626d6c0e72.slice/crio-601ce2c0ad40d3a234690a08421732098f4681e3fbaeb849a7ca38e38d35820c WatchSource:0}: Error finding container 601ce2c0ad40d3a234690a08421732098f4681e3fbaeb849a7ca38e38d35820c: Status 404 returned error can't find the container with id 601ce2c0ad40d3a234690a08421732098f4681e3fbaeb849a7ca38e38d35820c
Jul 24 13:40:23.694007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.693913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/884c3949-0d63-4988-9aa4-6a923533c921-manager-config\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.694007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.693977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-metrics-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.694007 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.694005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.694362 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.694037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hpr\" (UniqueName: \"kubernetes.io/projected/884c3949-0d63-4988-9aa4-6a923533c921-kube-api-access-27hpr\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.785027 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.784992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb" event={"ID":"feb7105e-55dc-4bf2-a756-22626d6c0e72","Type":"ContainerStarted","Data":"601ce2c0ad40d3a234690a08421732098f4681e3fbaeb849a7ca38e38d35820c"}
Jul 24 13:40:23.795434 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.795405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-metrics-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.795598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.795446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.795598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.795490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hpr\" (UniqueName: \"kubernetes.io/projected/884c3949-0d63-4988-9aa4-6a923533c921-kube-api-access-27hpr\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.795598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.795535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/884c3949-0d63-4988-9aa4-6a923533c921-manager-config\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.796224 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.796205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/884c3949-0d63-4988-9aa4-6a923533c921-manager-config\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.797995 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.797973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.798115 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.798095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/884c3949-0d63-4988-9aa4-6a923533c921-metrics-cert\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.809360 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.809334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hpr\" (UniqueName: \"kubernetes.io/projected/884c3949-0d63-4988-9aa4-6a923533c921-kube-api-access-27hpr\") pod \"lws-controller-manager-5d64c44479-p6hq5\" (UID: \"884c3949-0d63-4988-9aa4-6a923533c921\") " pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:23.912465 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:23.912426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:24.060188 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:24.060114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"]
Jul 24 13:40:24.063077 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:40:24.063052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884c3949_0d63_4988_9aa4_6a923533c921.slice/crio-82cc6488b6b546913b8977068f32f4e55062a17a9e76eb3588e23199f26df60e WatchSource:0}: Error finding container 82cc6488b6b546913b8977068f32f4e55062a17a9e76eb3588e23199f26df60e: Status 404 returned error can't find the container with id 82cc6488b6b546913b8977068f32f4e55062a17a9e76eb3588e23199f26df60e
Jul 24 13:40:24.790808 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:24.790769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5" event={"ID":"884c3949-0d63-4988-9aa4-6a923533c921","Type":"ContainerStarted","Data":"82cc6488b6b546913b8977068f32f4e55062a17a9e76eb3588e23199f26df60e"}
Jul 24 13:40:26.801252 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:26.801213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb" event={"ID":"feb7105e-55dc-4bf2-a756-22626d6c0e72","Type":"ContainerStarted","Data":"8739841461a24b73ebbd5ea5925ec9643fe2080ee4657a77d63aa96e376fc880"}
Jul 24 13:40:26.802726 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:26.802703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5" event={"ID":"884c3949-0d63-4988-9aa4-6a923533c921","Type":"ContainerStarted","Data":"5d630ee6956736a64d078a5762f080f532fdbbcf3aefec3d86b2186096ca06b1"}
Jul 24 13:40:26.802830 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:26.802749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:40:26.819618 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:26.819566    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85dc78bf9-jflxb" podStartSLOduration=1.199984128 podStartE2EDuration="3.819550485s" totalImagesPullingTime="2.619566357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:26.818271729 +0000 UTC m=+366.280878595" watchObservedRunningTime="2026-07-24 13:40:26.819550485 +0000 UTC m=+366.282157328"
Jul 24 13:40:26.836125 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:26.836077    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5" podStartSLOduration=1.6479122309999998 podStartE2EDuration="3.836061066s" totalImagesPullingTime="2.188148835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:26.835381589 +0000 UTC m=+366.297988456" watchObservedRunningTime="2026-07-24 13:40:26.836061066 +0000 UTC m=+366.298667910"
Jul 24 13:40:37.809542 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:40:37.809511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d64c44479-p6hq5"
Jul 24 13:41:58.924177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:58.924143    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57c658fdc4-944s4"]
Jul 24 13:41:58.926412 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:58.926395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:58.966842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:58.966806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c658fdc4-944s4"]
Jul 24 13:41:59.088223 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-console-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088223 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-oauth-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088430 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-service-ca\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088430 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-trusted-ca-bundle\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088430 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088567 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-oauth-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.088567 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.088466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hb5\" (UniqueName: \"kubernetes.io/projected/1366fb54-56f9-41cc-b790-2410878f0bbc-kube-api-access-s2hb5\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189656 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-service-ca\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189656 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-trusted-ca-bundle\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189656 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-oauth-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hb5\" (UniqueName: \"kubernetes.io/projected/1366fb54-56f9-41cc-b790-2410878f0bbc-kube-api-access-s2hb5\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-console-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.189921 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.189782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-oauth-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.190414 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.190337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-service-ca\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.190414 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.190400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-oauth-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.190716 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.190532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-console-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.190797 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.190772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1366fb54-56f9-41cc-b790-2410878f0bbc-trusted-ca-bundle\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.192272 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.192249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-serving-cert\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.192361 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.192289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1366fb54-56f9-41cc-b790-2410878f0bbc-console-oauth-config\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.207211 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.207188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hb5\" (UniqueName: \"kubernetes.io/projected/1366fb54-56f9-41cc-b790-2410878f0bbc-kube-api-access-s2hb5\") pod \"console-57c658fdc4-944s4\" (UID: \"1366fb54-56f9-41cc-b790-2410878f0bbc\") " pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.235260 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.235226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:41:59.383331 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:41:59.383307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57c658fdc4-944s4"]
Jul 24 13:41:59.385503 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:41:59.385466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1366fb54_56f9_41cc_b790_2410878f0bbc.slice/crio-719c87a0f50e395a307120f8cff20a694c09aaae9b475ed53776daf4b1e758d3 WatchSource:0}: Error finding container 719c87a0f50e395a307120f8cff20a694c09aaae9b475ed53776daf4b1e758d3: Status 404 returned error can't find the container with id 719c87a0f50e395a307120f8cff20a694c09aaae9b475ed53776daf4b1e758d3
Jul 24 13:42:00.183246 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:00.183214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c658fdc4-944s4" event={"ID":"1366fb54-56f9-41cc-b790-2410878f0bbc","Type":"ContainerStarted","Data":"ca53869fef19ea8fbf0b1af0fc8020dddbf76a8e2ac968e65dde834df8aa1e6e"}
Jul 24 13:42:00.183656 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:00.183287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57c658fdc4-944s4" event={"ID":"1366fb54-56f9-41cc-b790-2410878f0bbc","Type":"ContainerStarted","Data":"719c87a0f50e395a307120f8cff20a694c09aaae9b475ed53776daf4b1e758d3"}
Jul 24 13:42:00.205849 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:00.205800    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57c658fdc4-944s4" podStartSLOduration=2.205783623 podStartE2EDuration="2.205783623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:00.203960772 +0000 UTC m=+459.666567617" watchObservedRunningTime="2026-07-24 13:42:00.205783623 +0000 UTC m=+459.668390468"
Jul 24 13:42:04.476938 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.476901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"]
Jul 24 13:42:04.479583 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.479565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:04.482296 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.482251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 13:42:04.482514 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.482411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 13:42:04.483438 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.483413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s5c2r\""
Jul 24 13:42:04.496925 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.496897    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"]
Jul 24 13:42:04.642404 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.642371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngn5\" (UniqueName: \"kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5\") pod \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" (UID: \"2abe96f2-0307-408e-a2d0-fb4db257f07d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:04.743134 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.743048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngn5\" (UniqueName: \"kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5\") pod \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" (UID: \"2abe96f2-0307-408e-a2d0-fb4db257f07d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:04.752798 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.752767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngn5\" (UniqueName: \"kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5\") pod \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" (UID: \"2abe96f2-0307-408e-a2d0-fb4db257f07d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:04.790410 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.790377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:04.931370 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:04.931341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"]
Jul 24 13:42:04.934995 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:42:04.934965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abe96f2_0307_408e_a2d0_fb4db257f07d.slice/crio-2a0fd8a7b675c5489c346f6cc51e98781e78e40cfae2fb072f63acc6f1c4673a WatchSource:0}: Error finding container 2a0fd8a7b675c5489c346f6cc51e98781e78e40cfae2fb072f63acc6f1c4673a: Status 404 returned error can't find the container with id 2a0fd8a7b675c5489c346f6cc51e98781e78e40cfae2fb072f63acc6f1c4673a
Jul 24 13:42:05.204923 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:05.204884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt" event={"ID":"2abe96f2-0307-408e-a2d0-fb4db257f07d","Type":"ContainerStarted","Data":"2a0fd8a7b675c5489c346f6cc51e98781e78e40cfae2fb072f63acc6f1c4673a"}
Jul 24 13:42:07.216505 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:07.216385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt" event={"ID":"2abe96f2-0307-408e-a2d0-fb4db257f07d","Type":"ContainerStarted","Data":"c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad"}
Jul 24 13:42:07.216842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:07.216521    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:07.244903 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:07.244854    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt" podStartSLOduration=1.32223743 podStartE2EDuration="3.244839119s" totalImagesPullingTime="1.922601689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:07.244622672 +0000 UTC m=+466.707229516" watchObservedRunningTime="2026-07-24 13:42:07.244839119 +0000 UTC m=+466.707446038"
Jul 24 13:42:09.235646 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:09.235616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:42:09.236078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:09.235675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:42:09.240724 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:09.240703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:42:10.232554 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:10.232527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57c658fdc4-944s4"
Jul 24 13:42:10.287126 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:10.287095    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:42:11.358194 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.358157    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"]
Jul 24 13:42:11.360346 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.360330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.363003 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.362980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 24 13:42:11.363106 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.362993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 24 13:42:11.363106 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.363015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wbnz6\""
Jul 24 13:42:11.369297 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.369274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"]
Jul 24 13:42:11.403776 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.403739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c1d197-bad6-4963-a86a-30f31a209818-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.403882 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.403820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfj8\" (UniqueName: \"kubernetes.io/projected/40c1d197-bad6-4963-a86a-30f31a209818-kube-api-access-4pfj8\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.403882 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.403851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40c1d197-bad6-4963-a86a-30f31a209818-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.504255 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.504218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c1d197-bad6-4963-a86a-30f31a209818-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.504420 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.504307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfj8\" (UniqueName: \"kubernetes.io/projected/40c1d197-bad6-4963-a86a-30f31a209818-kube-api-access-4pfj8\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.504420 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.504351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40c1d197-bad6-4963-a86a-30f31a209818-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.505078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.505049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/40c1d197-bad6-4963-a86a-30f31a209818-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.506786 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.506765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/40c1d197-bad6-4963-a86a-30f31a209818-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.513874 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.513852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfj8\" (UniqueName: \"kubernetes.io/projected/40c1d197-bad6-4963-a86a-30f31a209818-kube-api-access-4pfj8\") pod \"kuadrant-console-plugin-66b5774985-b8ksv\" (UID: \"40c1d197-bad6-4963-a86a-30f31a209818\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.669813 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.669726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"
Jul 24 13:42:11.799383 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:11.799358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv"]
Jul 24 13:42:11.801566 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:42:11.801539    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40c1d197_bad6_4963_a86a_30f31a209818.slice/crio-ad9094de865c5b8b438d82ae73de38f38f9b4e6b0d46ecd7796dcf1359499281 WatchSource:0}: Error finding container ad9094de865c5b8b438d82ae73de38f38f9b4e6b0d46ecd7796dcf1359499281: Status 404 returned error can't find the container with id ad9094de865c5b8b438d82ae73de38f38f9b4e6b0d46ecd7796dcf1359499281
Jul 24 13:42:12.238651 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:12.238618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv" event={"ID":"40c1d197-bad6-4963-a86a-30f31a209818","Type":"ContainerStarted","Data":"ad9094de865c5b8b438d82ae73de38f38f9b4e6b0d46ecd7796dcf1359499281"}
Jul 24 13:42:18.222696 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:18.222622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:21.715177 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.715127    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"]
Jul 24 13:42:21.715692 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.715309    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" containerName="manager" containerID="cri-o://c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad" gracePeriod=2
Jul 24 13:42:21.722533 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.722501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"]
Jul 24 13:42:21.744279 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.744243    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"]
Jul 24 13:42:21.744685 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.744653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" containerName="manager"
Jul 24 13:42:21.744685 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.744675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" containerName="manager"
Jul 24 13:42:21.744876 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.744750    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" containerName="manager"
Jul 24 13:42:21.747796 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.747773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:21.750490 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:21.750444    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:21.766134 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.766090    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"]
Jul 24 13:42:21.795897 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.795862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5t6\" (UniqueName: \"kubernetes.io/projected/75b2e2b1-7cce-4e32-99ab-a203716b8bab-kube-api-access-qb5t6\") pod \"limitador-operator-controller-manager-859d7b78bd-nwdnl\" (UID: \"75b2e2b1-7cce-4e32-99ab-a203716b8bab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:21.896897 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.896856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5t6\" (UniqueName: \"kubernetes.io/projected/75b2e2b1-7cce-4e32-99ab-a203716b8bab-kube-api-access-qb5t6\") pod \"limitador-operator-controller-manager-859d7b78bd-nwdnl\" (UID: \"75b2e2b1-7cce-4e32-99ab-a203716b8bab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:21.910273 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:21.910239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5t6\" (UniqueName: \"kubernetes.io/projected/75b2e2b1-7cce-4e32-99ab-a203716b8bab-kube-api-access-qb5t6\") pod \"limitador-operator-controller-manager-859d7b78bd-nwdnl\" (UID: \"75b2e2b1-7cce-4e32-99ab-a203716b8bab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:22.104551 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:22.104465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:33.978105 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:33.978071    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"]
Jul 24 13:42:33.981738 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:42:33.981694    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b2e2b1_7cce_4e32_99ab_a203716b8bab.slice/crio-dd2b602db04aef270a876c6d09250ab7efb1ee26bd9583f82d0c697afde4b178 WatchSource:0}: Error finding container dd2b602db04aef270a876c6d09250ab7efb1ee26bd9583f82d0c697afde4b178: Status 404 returned error can't find the container with id dd2b602db04aef270a876c6d09250ab7efb1ee26bd9583f82d0c697afde4b178
Jul 24 13:42:34.089165 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.089137    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:34.091728 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:34.091658    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:34.208409 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.208375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tngn5\" (UniqueName: \"kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5\") pod \"2abe96f2-0307-408e-a2d0-fb4db257f07d\" (UID: \"2abe96f2-0307-408e-a2d0-fb4db257f07d\") "
Jul 24 13:42:34.210753 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.210728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5" (OuterVolumeSpecName: "kube-api-access-tngn5") pod "2abe96f2-0307-408e-a2d0-fb4db257f07d" (UID: "2abe96f2-0307-408e-a2d0-fb4db257f07d"). InnerVolumeSpecName "kube-api-access-tngn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:34.309463 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.309437    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tngn5\" (UniqueName: \"kubernetes.io/projected/2abe96f2-0307-408e-a2d0-fb4db257f07d-kube-api-access-tngn5\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:34.359235 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.359167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl" event={"ID":"75b2e2b1-7cce-4e32-99ab-a203716b8bab","Type":"ContainerStarted","Data":"78b79ab9caffc1cc50698141bca824c997b5ad694bcb2ced06d77073842038a8"}
Jul 24 13:42:34.359235 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.359224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl" event={"ID":"75b2e2b1-7cce-4e32-99ab-a203716b8bab","Type":"ContainerStarted","Data":"dd2b602db04aef270a876c6d09250ab7efb1ee26bd9583f82d0c697afde4b178"}
Jul 24 13:42:34.359436 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.359269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:34.360405 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.360376    2579 generic.go:356] "Generic (PLEG): container finished" podID="2abe96f2-0307-408e-a2d0-fb4db257f07d" containerID="c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad" exitCode=0
Jul 24 13:42:34.360405 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.360398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:34.360583 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.360434    2579 scope.go:117] "RemoveContainer" containerID="c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad"
Jul 24 13:42:34.361614 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:34.361593    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:34.369332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.369310    2579 scope.go:117] "RemoveContainer" containerID="c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad"
Jul 24 13:42:34.369606 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:34.369589    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad\": container with ID starting with c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad not found: ID does not exist" containerID="c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad"
Jul 24 13:42:34.369663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.369615    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad"} err="failed to get container status \"c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad\": rpc error: code = NotFound desc = could not find container \"c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad\": container with ID starting with c57183b8f96360e048cf1ea9230e93175aa1a6d0f9c273692c5d6da481c4e4ad not found: ID does not exist"
Jul 24 13:42:34.386571 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:34.386522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl" podStartSLOduration=13.38650706 podStartE2EDuration="13.38650706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:34.385155864 +0000 UTC m=+493.847762700" watchObservedRunningTime="2026-07-24 13:42:34.38650706 +0000 UTC m=+493.849113881"
Jul 24 13:42:34.387364 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:34.387342    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kcfbt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kcfbt"
Jul 24 13:42:35.124098 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.124024    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2abe96f2-0307-408e-a2d0-fb4db257f07d" path="/var/lib/kubelet/pods/2abe96f2-0307-408e-a2d0-fb4db257f07d/volumes"
Jul 24 13:42:35.306573 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.306525    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-847b995479-rt8xp" podUID="b07e15d9-321f-4930-8fa3-bf2a012285df" containerName="console" containerID="cri-o://c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7" gracePeriod=15
Jul 24 13:42:35.367218 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.367178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv" event={"ID":"40c1d197-bad6-4963-a86a-30f31a209818","Type":"ContainerStarted","Data":"a060c7729d03eb35ed6a15fe752ab2aded56215842cad2dee63abc2dc871db8c"}
Jul 24 13:42:35.383920 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.383834    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b8ksv" podStartSLOduration=1.910457057 podStartE2EDuration="24.383820109s" totalImagesPullingTime="22.473363052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:35.382601509 +0000 UTC m=+494.845208361" watchObservedRunningTime="2026-07-24 13:42:35.383820109 +0000 UTC m=+494.846426952"
Jul 24 13:42:35.583709 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.583678    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:42:35.721730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.721730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.721730 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721712    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.722010 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721741    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.722010 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssfbw\" (UniqueName: \"kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.722010 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.722010 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.721853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert\") pod \"b07e15d9-321f-4930-8fa3-bf2a012285df\" (UID: \"b07e15d9-321f-4930-8fa3-bf2a012285df\") "
Jul 24 13:42:35.722210 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.722089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:35.722274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.722247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config" (OuterVolumeSpecName: "console-config") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:35.722324 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.722150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:35.722445 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.722420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca" (OuterVolumeSpecName: "service-ca") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 13:42:35.724153 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.724123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:42:35.724249 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.724230    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:42:35.724295 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.724256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw" (OuterVolumeSpecName: "kube-api-access-ssfbw") pod "b07e15d9-321f-4930-8fa3-bf2a012285df" (UID: "b07e15d9-321f-4930-8fa3-bf2a012285df"). InnerVolumeSpecName "kube-api-access-ssfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:42:35.822851 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822815    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-trusted-ca-bundle\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.822851 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822845    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-oauth-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.822851 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822855    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.823085 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822864    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b07e15d9-321f-4930-8fa3-bf2a012285df-console-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.823085 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822872    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfbw\" (UniqueName: \"kubernetes.io/projected/b07e15d9-321f-4930-8fa3-bf2a012285df-kube-api-access-ssfbw\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.823085 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822881    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-oauth-config\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:35.823085 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:35.822889    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b07e15d9-321f-4930-8fa3-bf2a012285df-console-serving-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:42:36.372769 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.372729    2579 generic.go:356] "Generic (PLEG): container finished" podID="b07e15d9-321f-4930-8fa3-bf2a012285df" containerID="c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7" exitCode=0
Jul 24 13:42:36.373225 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.372775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b995479-rt8xp"
Jul 24 13:42:36.373225 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.372822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b995479-rt8xp" event={"ID":"b07e15d9-321f-4930-8fa3-bf2a012285df","Type":"ContainerDied","Data":"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"}
Jul 24 13:42:36.373225 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.372882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b995479-rt8xp" event={"ID":"b07e15d9-321f-4930-8fa3-bf2a012285df","Type":"ContainerDied","Data":"4c9d136e4eaa747c7dc85fcee5511668a94b7b20b90f08f20bda07539bb8541d"}
Jul 24 13:42:36.373225 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.372913    2579 scope.go:117] "RemoveContainer" containerID="c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"
Jul 24 13:42:36.382335 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.382319    2579 scope.go:117] "RemoveContainer" containerID="c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"
Jul 24 13:42:36.382630 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:42:36.382612    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7\": container with ID starting with c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7 not found: ID does not exist" containerID="c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"
Jul 24 13:42:36.382681 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.382640    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7"} err="failed to get container status \"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7\": rpc error: code = NotFound desc = could not find container \"c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7\": container with ID starting with c660cff47859760b496565099a8ee723589a5960da1e7b9c7807e193cf49a4b7 not found: ID does not exist"
Jul 24 13:42:36.395076 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.395052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:42:36.401443 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:36.401423    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-847b995479-rt8xp"]
Jul 24 13:42:37.124296 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:37.124264    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07e15d9-321f-4930-8fa3-bf2a012285df" path="/var/lib/kubelet/pods/b07e15d9-321f-4930-8fa3-bf2a012285df/volumes"
Jul 24 13:42:45.369429 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:45.369397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nwdnl"
Jul 24 13:42:53.897558 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.897524    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:42:53.898012 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.897878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07e15d9-321f-4930-8fa3-bf2a012285df" containerName="console"
Jul 24 13:42:53.898012 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.897891    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07e15d9-321f-4930-8fa3-bf2a012285df" containerName="console"
Jul 24 13:42:53.898012 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.897961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07e15d9-321f-4930-8fa3-bf2a012285df" containerName="console"
Jul 24 13:42:53.900912 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.900895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:42:53.903437 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.903417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-52l8r\""
Jul 24 13:42:53.913074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.913052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:42:53.984584 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:53.984553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkft\" (UniqueName: \"kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft\") pod \"authorino-58cd955675-llbrt\" (UID: \"87d43214-4b37-4045-8e7e-8a9fd09e5cc4\") " pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:42:54.085286 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:54.085256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkft\" (UniqueName: \"kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft\") pod \"authorino-58cd955675-llbrt\" (UID: \"87d43214-4b37-4045-8e7e-8a9fd09e5cc4\") " pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:42:54.094329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:54.094309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkft\" (UniqueName: \"kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft\") pod \"authorino-58cd955675-llbrt\" (UID: \"87d43214-4b37-4045-8e7e-8a9fd09e5cc4\") " pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:42:54.209919 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:54.209836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:42:54.337931 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:54.337906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:42:54.340529 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:42:54.340506    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87d43214_4b37_4045_8e7e_8a9fd09e5cc4.slice/crio-820406bd296f3e3c57544f90f3bbd94c10cf7a7d18986cee52b8fa215566475c WatchSource:0}: Error finding container 820406bd296f3e3c57544f90f3bbd94c10cf7a7d18986cee52b8fa215566475c: Status 404 returned error can't find the container with id 820406bd296f3e3c57544f90f3bbd94c10cf7a7d18986cee52b8fa215566475c
Jul 24 13:42:54.455657 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:54.455627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-llbrt" event={"ID":"87d43214-4b37-4045-8e7e-8a9fd09e5cc4","Type":"ContainerStarted","Data":"820406bd296f3e3c57544f90f3bbd94c10cf7a7d18986cee52b8fa215566475c"}
Jul 24 13:42:58.476817 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:58.476783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-llbrt" event={"ID":"87d43214-4b37-4045-8e7e-8a9fd09e5cc4","Type":"ContainerStarted","Data":"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"}
Jul 24 13:42:58.493332 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:58.493279    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-llbrt" podStartSLOduration=2.4241069729999998 podStartE2EDuration="5.493263598s" totalImagesPullingTime="3.069156625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:58.491804825 +0000 UTC m=+517.954411670" watchObservedRunningTime="2026-07-24 13:42:58.493263598 +0000 UTC m=+517.955870442"
Jul 24 13:42:59.169588 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:42:59.169557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:43:00.486711 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:00.486656    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-llbrt" podUID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" containerName="authorino" containerID="cri-o://5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233" gracePeriod=30
Jul 24 13:43:00.760540 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:00.760508    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:43:00.842642 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:00.842611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btkft\" (UniqueName: \"kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft\") pod \"87d43214-4b37-4045-8e7e-8a9fd09e5cc4\" (UID: \"87d43214-4b37-4045-8e7e-8a9fd09e5cc4\") "
Jul 24 13:43:00.845037 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:00.845001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft" (OuterVolumeSpecName: "kube-api-access-btkft") pod "87d43214-4b37-4045-8e7e-8a9fd09e5cc4" (UID: "87d43214-4b37-4045-8e7e-8a9fd09e5cc4"). InnerVolumeSpecName "kube-api-access-btkft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:00.944106 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:00.944073    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkft\" (UniqueName: \"kubernetes.io/projected/87d43214-4b37-4045-8e7e-8a9fd09e5cc4-kube-api-access-btkft\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:43:01.492535 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.492427    2579 generic.go:356] "Generic (PLEG): container finished" podID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" containerID="5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233" exitCode=0
Jul 24 13:43:01.492535 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.492512    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-llbrt"
Jul 24 13:43:01.492535 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.492519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-llbrt" event={"ID":"87d43214-4b37-4045-8e7e-8a9fd09e5cc4","Type":"ContainerDied","Data":"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"}
Jul 24 13:43:01.493097 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.492563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-llbrt" event={"ID":"87d43214-4b37-4045-8e7e-8a9fd09e5cc4","Type":"ContainerDied","Data":"820406bd296f3e3c57544f90f3bbd94c10cf7a7d18986cee52b8fa215566475c"}
Jul 24 13:43:01.493097 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.492580    2579 scope.go:117] "RemoveContainer" containerID="5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"
Jul 24 13:43:01.501028 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.501010    2579 scope.go:117] "RemoveContainer" containerID="5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"
Jul 24 13:43:01.501303 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:43:01.501284    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233\": container with ID starting with 5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233 not found: ID does not exist" containerID="5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"
Jul 24 13:43:01.501355 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.501312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233"} err="failed to get container status \"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233\": rpc error: code = NotFound desc = could not find container \"5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233\": container with ID starting with 5f69358a81a0b3fffa9a8db9d7a93b2700f7ce30ccd56ab466eaa755f9e4a233 not found: ID does not exist"
Jul 24 13:43:01.508993 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.508969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:43:01.517585 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:01.517559    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-llbrt"]
Jul 24 13:43:03.123494 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:03.123453    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" path="/var/lib/kubelet/pods/87d43214-4b37-4045-8e7e-8a9fd09e5cc4/volumes"
Jul 24 13:43:21.831455 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.831423    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:21.831864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.831743    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" containerName="authorino"
Jul 24 13:43:21.831864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.831755    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" containerName="authorino"
Jul 24 13:43:21.831864 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.831808    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d43214-4b37-4045-8e7e-8a9fd09e5cc4" containerName="authorino"
Jul 24 13:43:21.834308 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.834290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:21.838875 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.838855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-52l8r\""
Jul 24 13:43:21.853834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.853807    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:21.931607 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:21.931571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np5np\" (UniqueName: \"kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np\") pod \"authorino-5d4b67c446-wpd5f\" (UID: \"310bbb47-8246-4ffc-b7c8-6b534d543012\") " pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:22.032285 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.032242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np5np\" (UniqueName: \"kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np\") pod \"authorino-5d4b67c446-wpd5f\" (UID: \"310bbb47-8246-4ffc-b7c8-6b534d543012\") " pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:22.041721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.041696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np5np\" (UniqueName: \"kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np\") pod \"authorino-5d4b67c446-wpd5f\" (UID: \"310bbb47-8246-4ffc-b7c8-6b534d543012\") " pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:22.066156 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.066119    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:22.066374 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.066362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:22.200081 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.200011    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:22.202885 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:43:22.202856    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod310bbb47_8246_4ffc_b7c8_6b534d543012.slice/crio-a99418890fe8b09ff4a035abca755c4ca89a092cf20202694645d24c1aff54c3 WatchSource:0}: Error finding container a99418890fe8b09ff4a035abca755c4ca89a092cf20202694645d24c1aff54c3: Status 404 returned error can't find the container with id a99418890fe8b09ff4a035abca755c4ca89a092cf20202694645d24c1aff54c3
Jul 24 13:43:22.496333 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.496235    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:43:22.499180 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.499163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.501839 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.501814    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 13:43:22.507199 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.507173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:43:22.585858 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.585824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" event={"ID":"310bbb47-8246-4ffc-b7c8-6b534d543012","Type":"ContainerStarted","Data":"a99418890fe8b09ff4a035abca755c4ca89a092cf20202694645d24c1aff54c3"}
Jul 24 13:43:22.636747 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.636721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.636833 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.636802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9rf\" (UniqueName: \"kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.737444 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.737411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9rf\" (UniqueName: \"kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.737635 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.737538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.740108 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.740083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.746385 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.746330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9rf\" (UniqueName: \"kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf\") pod \"authorino-7b689779d5-9m5jb\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") " pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.809394 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.809350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:43:22.939265 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:22.939235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:43:22.942129 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:43:22.942106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1286ad2b_411b_40ff_a61f_7071e621242c.slice/crio-855559058ed45181b47b716cce25f5ae64aaeb767d5d8caf00f050576616487d WatchSource:0}: Error finding container 855559058ed45181b47b716cce25f5ae64aaeb767d5d8caf00f050576616487d: Status 404 returned error can't find the container with id 855559058ed45181b47b716cce25f5ae64aaeb767d5d8caf00f050576616487d
Jul 24 13:43:23.592011 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.591977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" event={"ID":"310bbb47-8246-4ffc-b7c8-6b534d543012","Type":"ContainerStarted","Data":"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"}
Jul 24 13:43:23.592215 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.592050    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" podUID="310bbb47-8246-4ffc-b7c8-6b534d543012" containerName="authorino" containerID="cri-o://1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e" gracePeriod=30
Jul 24 13:43:23.593858 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.593830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b689779d5-9m5jb" event={"ID":"1286ad2b-411b-40ff-a61f-7071e621242c","Type":"ContainerStarted","Data":"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"}
Jul 24 13:43:23.593967 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.593868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b689779d5-9m5jb" event={"ID":"1286ad2b-411b-40ff-a61f-7071e621242c","Type":"ContainerStarted","Data":"855559058ed45181b47b716cce25f5ae64aaeb767d5d8caf00f050576616487d"}
Jul 24 13:43:23.607900 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.607859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" podStartSLOduration=2.183866988 podStartE2EDuration="2.607843198s" totalImagesPullingTime="423.97621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:23.607508009 +0000 UTC m=+543.070114855" watchObservedRunningTime="2026-07-24 13:43:23.607843198 +0000 UTC m=+543.070450042"
Jul 24 13:43:23.623298 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.623110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b689779d5-9m5jb" podStartSLOduration=1.19677267 podStartE2EDuration="1.623095336s" totalImagesPullingTime="426.322666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:23.622816078 +0000 UTC m=+543.085422924" watchObservedRunningTime="2026-07-24 13:43:23.623095336 +0000 UTC m=+543.085702160"
Jul 24 13:43:23.819825 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.819799    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:23.947700 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.947604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np5np\" (UniqueName: \"kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np\") pod \"310bbb47-8246-4ffc-b7c8-6b534d543012\" (UID: \"310bbb47-8246-4ffc-b7c8-6b534d543012\") "
Jul 24 13:43:23.949785 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:23.949757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np" (OuterVolumeSpecName: "kube-api-access-np5np") pod "310bbb47-8246-4ffc-b7c8-6b534d543012" (UID: "310bbb47-8246-4ffc-b7c8-6b534d543012"). InnerVolumeSpecName "kube-api-access-np5np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:43:24.049135 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.049099    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np5np\" (UniqueName: \"kubernetes.io/projected/310bbb47-8246-4ffc-b7c8-6b534d543012-kube-api-access-np5np\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:43:24.599738 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.599696    2579 generic.go:356] "Generic (PLEG): container finished" podID="310bbb47-8246-4ffc-b7c8-6b534d543012" containerID="1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e" exitCode=0
Jul 24 13:43:24.599738 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.599738    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wpd5f"
Jul 24 13:43:24.599989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.599789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" event={"ID":"310bbb47-8246-4ffc-b7c8-6b534d543012","Type":"ContainerDied","Data":"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"}
Jul 24 13:43:24.599989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.599835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wpd5f" event={"ID":"310bbb47-8246-4ffc-b7c8-6b534d543012","Type":"ContainerDied","Data":"a99418890fe8b09ff4a035abca755c4ca89a092cf20202694645d24c1aff54c3"}
Jul 24 13:43:24.599989 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.599858    2579 scope.go:117] "RemoveContainer" containerID="1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"
Jul 24 13:43:24.608378 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.608361    2579 scope.go:117] "RemoveContainer" containerID="1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"
Jul 24 13:43:24.608685 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:43:24.608667    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e\": container with ID starting with 1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e not found: ID does not exist" containerID="1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"
Jul 24 13:43:24.608732 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.608694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e"} err="failed to get container status \"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e\": rpc error: code = NotFound desc = could not find container \"1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e\": container with ID starting with 1fe67a918ad4eeaeacbb4dde39dca43740bdebd511c8e26c0c4a686540d7ab2e not found: ID does not exist"
Jul 24 13:43:24.620869 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.620845    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:24.626173 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:24.626147    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wpd5f"]
Jul 24 13:43:25.124982 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:43:25.124947    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310bbb47-8246-4ffc-b7c8-6b534d543012" path="/var/lib/kubelet/pods/310bbb47-8246-4ffc-b7c8-6b534d543012/volumes"
Jul 24 13:44:20.972736 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:20.972709    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:44:20.973352 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:20.973332    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:44:39.101768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.101737    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"]
Jul 24 13:44:39.102182 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.102041    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310bbb47-8246-4ffc-b7c8-6b534d543012" containerName="authorino"
Jul 24 13:44:39.102182 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.102052    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="310bbb47-8246-4ffc-b7c8-6b534d543012" containerName="authorino"
Jul 24 13:44:39.102182 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.102123    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="310bbb47-8246-4ffc-b7c8-6b534d543012" containerName="authorino"
Jul 24 13:44:39.106318 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.106303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.109061 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.109044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 13:44:39.110183 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.110156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 13:44:39.110183 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.110174    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 13:44:39.110329 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.110192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2w2kd\""
Jul 24 13:44:39.125442 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.125417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"]
Jul 24 13:44:39.176720 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.176687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.176720 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.176721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.176918 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.176789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.177248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.176965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.177248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.177062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124a8b34-146c-430b-ab44-ef63952d1168-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.177248 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.177112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7249r\" (UniqueName: \"kubernetes.io/projected/124a8b34-146c-430b-ab44-ef63952d1168-kube-api-access-7249r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278070 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278070 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278318 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278318 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278318 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124a8b34-146c-430b-ab44-ef63952d1168-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278318 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7249r\" (UniqueName: \"kubernetes.io/projected/124a8b34-146c-430b-ab44-ef63952d1168-kube-api-access-7249r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278573 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278573 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.278649 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.278605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.280520 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.280499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124a8b34-146c-430b-ab44-ef63952d1168-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.280805 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.280787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124a8b34-146c-430b-ab44-ef63952d1168-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.302748 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.302718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7249r\" (UniqueName: \"kubernetes.io/projected/124a8b34-146c-430b-ab44-ef63952d1168-kube-api-access-7249r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfchr\" (UID: \"124a8b34-146c-430b-ab44-ef63952d1168\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.416997 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.416884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:39.550764 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.550732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"]
Jul 24 13:44:39.553999 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:44:39.553973    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124a8b34_146c_430b_ab44_ef63952d1168.slice/crio-5d2430837d0c2cf367e1f00e282879b4e31a72d512932a2c6e8b7cf5d6fd6080 WatchSource:0}: Error finding container 5d2430837d0c2cf367e1f00e282879b4e31a72d512932a2c6e8b7cf5d6fd6080: Status 404 returned error can't find the container with id 5d2430837d0c2cf367e1f00e282879b4e31a72d512932a2c6e8b7cf5d6fd6080
Jul 24 13:44:39.931905 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:39.931857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr" event={"ID":"124a8b34-146c-430b-ab44-ef63952d1168","Type":"ContainerStarted","Data":"5d2430837d0c2cf367e1f00e282879b4e31a72d512932a2c6e8b7cf5d6fd6080"}
Jul 24 13:44:40.938408 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:40.938319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr" event={"ID":"124a8b34-146c-430b-ab44-ef63952d1168","Type":"ContainerStarted","Data":"5e6dbbbf40ef8159e548a2c1caa0aa3218fc2f4a66b09d84b8ef04071c754ea7"}
Jul 24 13:44:40.957713 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:40.957664    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr" podStartSLOduration=0.830960841 podStartE2EDuration="1.957646115s" totalImagesPullingTime="1.126685274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:40.955754299 +0000 UTC m=+620.418361166" watchObservedRunningTime="2026-07-24 13:44:40.957646115 +0000 UTC m=+620.420252961"
Jul 24 13:44:44.229277 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.229246    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"]
Jul 24 13:44:44.232680 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.232659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.235676 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.235654    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 13:44:44.249777 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.249747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"]
Jul 24 13:44:44.323379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.323379 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d05bbeb-fc73-41f2-b926-657e5450d623-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.323663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.323663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.323663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.323663 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.323519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p2qj\" (UniqueName: \"kubernetes.io/projected/2d05bbeb-fc73-41f2-b926-657e5450d623-kube-api-access-9p2qj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424172 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424172 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p2qj\" (UniqueName: \"kubernetes.io/projected/2d05bbeb-fc73-41f2-b926-657e5450d623-kube-api-access-9p2qj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424425 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424425 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d05bbeb-fc73-41f2-b926-657e5450d623-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424425 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424425 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424667 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424709 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.424709 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.424691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.426689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.426668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d05bbeb-fc73-41f2-b926-657e5450d623-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.426910 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.426894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d05bbeb-fc73-41f2-b926-657e5450d623-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.434991 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.434970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p2qj\" (UniqueName: \"kubernetes.io/projected/2d05bbeb-fc73-41f2-b926-657e5450d623-kube-api-access-9p2qj\") pod \"e2e-trlp-test-simulated-kserve-598466548d-547k5\" (UID: \"2d05bbeb-fc73-41f2-b926-657e5450d623\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.542665 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.542631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:44.679493 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.679450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"]
Jul 24 13:44:44.681346 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:44:44.681320    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d05bbeb_fc73_41f2_b926_657e5450d623.slice/crio-c4b84a22d1ec19d024a666c72159a3bedf3257d3dd46902adce448f8a65c4390 WatchSource:0}: Error finding container c4b84a22d1ec19d024a666c72159a3bedf3257d3dd46902adce448f8a65c4390: Status 404 returned error can't find the container with id c4b84a22d1ec19d024a666c72159a3bedf3257d3dd46902adce448f8a65c4390
Jul 24 13:44:44.682809 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.682791    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 13:44:44.959969 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.959932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5" event={"ID":"2d05bbeb-fc73-41f2-b926-657e5450d623","Type":"ContainerStarted","Data":"a57d53a361b256dd6f0a16fb926f57d00c25275a21212972bb464c37ca0e2b13"}
Jul 24 13:44:44.960159 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.959986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5" event={"ID":"2d05bbeb-fc73-41f2-b926-657e5450d623","Type":"ContainerStarted","Data":"c4b84a22d1ec19d024a666c72159a3bedf3257d3dd46902adce448f8a65c4390"}
Jul 24 13:44:44.980772 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:44.980721    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5" podStartSLOduration=0.802464562 podStartE2EDuration="980.704275ms" totalImagesPullingTime="178.239713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:44.978520272 +0000 UTC m=+624.441127116" watchObservedRunningTime="2026-07-24 13:44:44.980704275 +0000 UTC m=+624.443311119"
Jul 24 13:44:49.417339 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:49.417291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:49.417845 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:49.417375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:49.429833 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:49.429812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:49.994932 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:49.994901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfchr"
Jul 24 13:44:54.543701 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:54.543662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:54.544078 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:54.543815    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:54.556087 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:54.556066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:44:55.016689 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:44:55.016662    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-547k5"
Jul 24 13:46:30.838181 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:30.838144    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65f757f4d8-qfq8x"]
Jul 24 13:46:30.841612 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:30.841594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:30.859852 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:30.859825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f757f4d8-qfq8x"]
Jul 24 13:46:30.972283 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:30.972242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e70d91-7dab-4d62-bdf2-acc562e92b7b-tls-cert\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:30.972467 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:30.972386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzfx\" (UniqueName: \"kubernetes.io/projected/16e70d91-7dab-4d62-bdf2-acc562e92b7b-kube-api-access-jwzfx\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.073313 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.073266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzfx\" (UniqueName: \"kubernetes.io/projected/16e70d91-7dab-4d62-bdf2-acc562e92b7b-kube-api-access-jwzfx\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.073526 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.073326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e70d91-7dab-4d62-bdf2-acc562e92b7b-tls-cert\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.076030 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.076001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/16e70d91-7dab-4d62-bdf2-acc562e92b7b-tls-cert\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.082742 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.082720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzfx\" (UniqueName: \"kubernetes.io/projected/16e70d91-7dab-4d62-bdf2-acc562e92b7b-kube-api-access-jwzfx\") pod \"authorino-65f757f4d8-qfq8x\" (UID: \"16e70d91-7dab-4d62-bdf2-acc562e92b7b\") " pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.150675 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.150589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65f757f4d8-qfq8x"
Jul 24 13:46:31.292943 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.292878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65f757f4d8-qfq8x"]
Jul 24 13:46:31.295677 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:46:31.295645    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16e70d91_7dab_4d62_bdf2_acc562e92b7b.slice/crio-f31063b4e1da08ee462fe2cd70488e1ab4216f3b61aa993a789b1a572329f9e1 WatchSource:0}: Error finding container f31063b4e1da08ee462fe2cd70488e1ab4216f3b61aa993a789b1a572329f9e1: Status 404 returned error can't find the container with id f31063b4e1da08ee462fe2cd70488e1ab4216f3b61aa993a789b1a572329f9e1
Jul 24 13:46:31.425589 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:31.425499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f757f4d8-qfq8x" event={"ID":"16e70d91-7dab-4d62-bdf2-acc562e92b7b","Type":"ContainerStarted","Data":"f31063b4e1da08ee462fe2cd70488e1ab4216f3b61aa993a789b1a572329f9e1"}
Jul 24 13:46:32.434002 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.433957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65f757f4d8-qfq8x" event={"ID":"16e70d91-7dab-4d62-bdf2-acc562e92b7b","Type":"ContainerStarted","Data":"35e150a487bd1c9131c7ddca8e37f562823b7ab2e40a8d6010731691bc5d7813"}
Jul 24 13:46:32.466658 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.466604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65f757f4d8-qfq8x" podStartSLOduration=2.058439084 podStartE2EDuration="2.466584821s" totalImagesPullingTime="408.145737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:32.465929662 +0000 UTC m=+731.928536507" watchObservedRunningTime="2026-07-24 13:46:32.466584821 +0000 UTC m=+731.929191718"
Jul 24 13:46:32.518700 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.518665    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:46:32.518896 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.518871    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7b689779d5-9m5jb" podUID="1286ad2b-411b-40ff-a61f-7071e621242c" containerName="authorino" containerID="cri-o://33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93" gracePeriod=30
Jul 24 13:46:32.777957 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.777930    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:46:32.889948 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.889913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf9rf\" (UniqueName: \"kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf\") pod \"1286ad2b-411b-40ff-a61f-7071e621242c\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") "
Jul 24 13:46:32.890107 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.889975    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert\") pod \"1286ad2b-411b-40ff-a61f-7071e621242c\" (UID: \"1286ad2b-411b-40ff-a61f-7071e621242c\") "
Jul 24 13:46:32.892301 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.892272    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf" (OuterVolumeSpecName: "kube-api-access-cf9rf") pod "1286ad2b-411b-40ff-a61f-7071e621242c" (UID: "1286ad2b-411b-40ff-a61f-7071e621242c"). InnerVolumeSpecName "kube-api-access-cf9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:46:32.901953 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.901922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "1286ad2b-411b-40ff-a61f-7071e621242c" (UID: "1286ad2b-411b-40ff-a61f-7071e621242c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:46:32.991394 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.991315    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9rf\" (UniqueName: \"kubernetes.io/projected/1286ad2b-411b-40ff-a61f-7071e621242c-kube-api-access-cf9rf\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:46:32.991394 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:32.991345    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1286ad2b-411b-40ff-a61f-7071e621242c-tls-cert\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:46:33.439774 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.439741    2579 generic.go:356] "Generic (PLEG): container finished" podID="1286ad2b-411b-40ff-a61f-7071e621242c" containerID="33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93" exitCode=0
Jul 24 13:46:33.440202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.439790    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b689779d5-9m5jb"
Jul 24 13:46:33.440202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.439832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b689779d5-9m5jb" event={"ID":"1286ad2b-411b-40ff-a61f-7071e621242c","Type":"ContainerDied","Data":"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"}
Jul 24 13:46:33.440202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.439878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b689779d5-9m5jb" event={"ID":"1286ad2b-411b-40ff-a61f-7071e621242c","Type":"ContainerDied","Data":"855559058ed45181b47b716cce25f5ae64aaeb767d5d8caf00f050576616487d"}
Jul 24 13:46:33.440202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.439902    2579 scope.go:117] "RemoveContainer" containerID="33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"
Jul 24 13:46:33.448721 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.448699    2579 scope.go:117] "RemoveContainer" containerID="33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"
Jul 24 13:46:33.448983 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:46:33.448964    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93\": container with ID starting with 33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93 not found: ID does not exist" containerID="33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"
Jul 24 13:46:33.449032 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.448991    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93"} err="failed to get container status \"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93\": rpc error: code = NotFound desc = could not find container \"33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93\": container with ID starting with 33931d3e5dd6a6e54c4c32f82b010467fe14939756b08d1ce9eeb07af0a2df93 not found: ID does not exist"
Jul 24 13:46:33.457022 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.456997    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:46:33.460584 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:33.460562    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b689779d5-9m5jb"]
Jul 24 13:46:35.128639 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:46:35.128609    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1286ad2b-411b-40ff-a61f-7071e621242c" path="/var/lib/kubelet/pods/1286ad2b-411b-40ff-a61f-7071e621242c/volumes"
Jul 24 13:49:20.998044 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:49:20.998006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:49:21.001156 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:49:21.001137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:54:21.023968 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:54:21.023941    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:54:21.027704 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:54:21.027685    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:57:06.497263 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.497232    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:06.497768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.497565    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1286ad2b-411b-40ff-a61f-7071e621242c" containerName="authorino"
Jul 24 13:57:06.497768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.497578    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1286ad2b-411b-40ff-a61f-7071e621242c" containerName="authorino"
Jul 24 13:57:06.497768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.497640    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1286ad2b-411b-40ff-a61f-7071e621242c" containerName="authorino"
Jul 24 13:57:06.500568 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.500549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.506470 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.506448    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-db689548-kserve-self-signed-certs\""
Jul 24 13:57:06.588202 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.588370 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.588370 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrjx\" (UniqueName: \"kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.588466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.588466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.588466 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.588421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.594073 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.594048    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:06.689015 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.688978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689015 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689274 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrjx\" (UniqueName: \"kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689500 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689563 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.689679 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.689644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.691506 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.691462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.692178 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.692158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.699559 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.699540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrjx\" (UniqueName: \"kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx\") pod \"e2e-same-id-a-db689548-kserve-f7879877f-rm6dv\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") " pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.810030 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.809997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:06.974666 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:06.974642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:06.977554 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:57:06.977526    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c0d2fc_0d2a_42f2_b667_779ed28cb922.slice/crio-df33813e9d017ad320d62f9c250718c91c34d9d9d83f04efe8415287bc02d49e WatchSource:0}: Error finding container df33813e9d017ad320d62f9c250718c91c34d9d9d83f04efe8415287bc02d49e: Status 404 returned error can't find the container with id df33813e9d017ad320d62f9c250718c91c34d9d9d83f04efe8415287bc02d49e
Jul 24 13:57:07.287659 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:07.287617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" event={"ID":"c1c0d2fc-0d2a-42f2-b667-779ed28cb922","Type":"ContainerStarted","Data":"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"}
Jul 24 13:57:07.287659 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:07.287664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" event={"ID":"c1c0d2fc-0d2a-42f2-b667-779ed28cb922","Type":"ContainerStarted","Data":"df33813e9d017ad320d62f9c250718c91c34d9d9d83f04efe8415287bc02d49e"}
Jul 24 13:57:07.322299 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:07.322243    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" podStartSLOduration=1.322224705 podStartE2EDuration="1.322224705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:57:07.321512121 +0000 UTC m=+1366.784118966" watchObservedRunningTime="2026-07-24 13:57:07.322224705 +0000 UTC m=+1366.784831578"
Jul 24 13:57:09.715191 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.715156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:09.718358 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.718342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.721309 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.721288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-db689548-kserve-self-signed-certs\""
Jul 24 13:57:09.729376 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.729356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:09.819662 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwhx\" (UniqueName: \"kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.819838 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.819838 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.819838 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.819838 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.819838 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.819796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.920836 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.920836 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921028 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921028 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwhx\" (UniqueName: \"kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921028 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921028 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.920967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921288 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.921264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.921271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.921359 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.921321    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.923330 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.923309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.923628 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.923611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:09.932816 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:09.932798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwhx\" (UniqueName: \"kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx\") pod \"e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") " pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:10.028115 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:10.028087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:10.194295 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:10.194270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:10.196737 ip-10-0-136-88 kubenswrapper[2579]: W0724 13:57:10.196709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38668f34_08d9_4569_aeeb_46ff422e4cc3.slice/crio-5e0777460f22ec5ce0e23b9ffb22dc8a10de55672805c8285877e0d7d95ad092 WatchSource:0}: Error finding container 5e0777460f22ec5ce0e23b9ffb22dc8a10de55672805c8285877e0d7d95ad092: Status 404 returned error can't find the container with id 5e0777460f22ec5ce0e23b9ffb22dc8a10de55672805c8285877e0d7d95ad092
Jul 24 13:57:10.302653 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:10.302572    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" event={"ID":"38668f34-08d9-4569-aeeb-46ff422e4cc3","Type":"ContainerStarted","Data":"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"}
Jul 24 13:57:10.302653 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:10.302642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" event={"ID":"38668f34-08d9-4569-aeeb-46ff422e4cc3","Type":"ContainerStarted","Data":"5e0777460f22ec5ce0e23b9ffb22dc8a10de55672805c8285877e0d7d95ad092"}
Jul 24 13:57:10.339237 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:10.339189    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" podStartSLOduration=1.339172513 podStartE2EDuration="1.339172513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:57:10.338910506 +0000 UTC m=+1369.801517349" watchObservedRunningTime="2026-07-24 13:57:10.339172513 +0000 UTC m=+1369.801779356"
Jul 24 13:57:16.811064 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:16.811029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:16.811597 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:16.811105    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:16.827953 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:16.827925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:17.346229 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:17.346197    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:20.028714 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:20.028665    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:20.028714 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:20.028723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:20.041350 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:20.041326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:20.362577 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:20.361282    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:25.862113 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:25.862076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:27.060915 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:27.060883    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:40.881559 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:40.881517    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" podUID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" containerName="main" containerID="cri-o://db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef" gracePeriod=45
Jul 24 13:57:41.124599 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.124572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:41.311287 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrjx\" (UniqueName: \"kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311317    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311469 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311434    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir\") pod \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\" (UID: \"c1c0d2fc-0d2a-42f2-b667-779ed28cb922\") "
Jul 24 13:57:41.311834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home" (OuterVolumeSpecName: "home") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:41.311834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:41.311834 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.311750    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:41.313640 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.313619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm" (OuterVolumeSpecName: "dshm") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:41.313729 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.313672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx" (OuterVolumeSpecName: "kube-api-access-tnrjx") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "kube-api-access-tnrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:41.313798 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.313738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1c0d2fc-0d2a-42f2-b667-779ed28cb922" (UID: "c1c0d2fc-0d2a-42f2-b667-779ed28cb922"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:57:41.413074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413035    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrjx\" (UniqueName: \"kubernetes.io/projected/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-kube-api-access-tnrjx\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.413074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413066    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.413074 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413077    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.413309 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413086    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.413309 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413094    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.413309 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.413101    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c0d2fc-0d2a-42f2-b667-779ed28cb922-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:41.446253 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.446218    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" containerID="db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef" exitCode=0
Jul 24 13:57:41.446419 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.446266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"
Jul 24 13:57:41.446419 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.446309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" event={"ID":"c1c0d2fc-0d2a-42f2-b667-779ed28cb922","Type":"ContainerDied","Data":"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"}
Jul 24 13:57:41.446419 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.446354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv" event={"ID":"c1c0d2fc-0d2a-42f2-b667-779ed28cb922","Type":"ContainerDied","Data":"df33813e9d017ad320d62f9c250718c91c34d9d9d83f04efe8415287bc02d49e"}
Jul 24 13:57:41.446419 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.446371    2579 scope.go:117] "RemoveContainer" containerID="db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"
Jul 24 13:57:41.458846 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.458831    2579 scope.go:117] "RemoveContainer" containerID="db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"
Jul 24 13:57:41.459154 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:57:41.459137    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef\": container with ID starting with db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef not found: ID does not exist" containerID="db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"
Jul 24 13:57:41.459214 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.459166    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef"} err="failed to get container status \"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef\": rpc error: code = NotFound desc = could not find container \"db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef\": container with ID starting with db1f6d8316aa9a7635e2d1fcc745638b15eaaf955040de302367583cec9910ef not found: ID does not exist"
Jul 24 13:57:41.467842 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.467821    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:41.471750 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:41.471717    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-db689548-kserve-f7879877f-rm6dv"]
Jul 24 13:57:42.079647 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.079608    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" podUID="38668f34-08d9-4569-aeeb-46ff422e4cc3" containerName="main" containerID="cri-o://2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a" gracePeriod=45
Jul 24 13:57:42.329610 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.329583    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:42.458572 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.458463    2579 generic.go:356] "Generic (PLEG): container finished" podID="38668f34-08d9-4569-aeeb-46ff422e4cc3" containerID="2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a" exitCode=0
Jul 24 13:57:42.458572 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.458524    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"
Jul 24 13:57:42.458760 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.458576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" event={"ID":"38668f34-08d9-4569-aeeb-46ff422e4cc3","Type":"ContainerDied","Data":"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"}
Jul 24 13:57:42.458760 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.458613    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp" event={"ID":"38668f34-08d9-4569-aeeb-46ff422e4cc3","Type":"ContainerDied","Data":"5e0777460f22ec5ce0e23b9ffb22dc8a10de55672805c8285877e0d7d95ad092"}
Jul 24 13:57:42.458760 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.458630    2579 scope.go:117] "RemoveContainer" containerID="2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"
Jul 24 13:57:42.466891 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.466878    2579 scope.go:117] "RemoveContainer" containerID="2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"
Jul 24 13:57:42.467139 ip-10-0-136-88 kubenswrapper[2579]: E0724 13:57:42.467120    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a\": container with ID starting with 2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a not found: ID does not exist" containerID="2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"
Jul 24 13:57:42.467187 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.467148    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a"} err="failed to get container status \"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a\": rpc error: code = NotFound desc = could not find container \"2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a\": container with ID starting with 2387574646a36f7e577e03889d19c03472f28748c55c11c3a47a54f3baba101a not found: ID does not exist"
Jul 24 13:57:42.522598 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzwhx\" (UniqueName: \"kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522701 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522701 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522701 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522854 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522854 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm\") pod \"38668f34-08d9-4569-aeeb-46ff422e4cc3\" (UID: \"38668f34-08d9-4569-aeeb-46ff422e4cc3\") "
Jul 24 13:57:42.522996 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:42.522996 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home" (OuterVolumeSpecName: "home") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:42.522996 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.522975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:42.524899 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.524870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm" (OuterVolumeSpecName: "dshm") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 13:57:42.524987 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.524943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx" (OuterVolumeSpecName: "kube-api-access-xzwhx") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "kube-api-access-xzwhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 13:57:42.524987 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.524953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38668f34-08d9-4569-aeeb-46ff422e4cc3" (UID: "38668f34-08d9-4569-aeeb-46ff422e4cc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 13:57:42.624024 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.623984    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.624024 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.624014    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.624024 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.624024    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzwhx\" (UniqueName: \"kubernetes.io/projected/38668f34-08d9-4569-aeeb-46ff422e4cc3-kube-api-access-xzwhx\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.624024 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.624035    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38668f34-08d9-4569-aeeb-46ff422e4cc3-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.624273 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.624045    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.624273 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.624052    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38668f34-08d9-4569-aeeb-46ff422e4cc3-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 13:57:42.781034 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.781006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:42.786455 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:42.786428    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-db689548-kserve-6d64c8cf6c-txrzp"]
Jul 24 13:57:43.124616 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:43.124539    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38668f34-08d9-4569-aeeb-46ff422e4cc3" path="/var/lib/kubelet/pods/38668f34-08d9-4569-aeeb-46ff422e4cc3/volumes"
Jul 24 13:57:43.124996 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:57:43.124931    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" path="/var/lib/kubelet/pods/c1c0d2fc-0d2a-42f2-b667-779ed28cb922/volumes"
Jul 24 13:59:21.049645 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:59:21.049618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 13:59:21.054768 ip-10-0-136-88 kubenswrapper[2579]: I0724 13:59:21.054748    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:01:41.056368 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056335    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"]
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056718    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" containerName="main"
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056731    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" containerName="main"
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056746    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38668f34-08d9-4569-aeeb-46ff422e4cc3" containerName="main"
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38668f34-08d9-4569-aeeb-46ff422e4cc3" containerName="main"
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056798    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38668f34-08d9-4569-aeeb-46ff422e4cc3" containerName="main"
Jul 24 14:01:41.056831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.056806    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c0d2fc-0d2a-42f2-b667-779ed28cb922" containerName="main"
Jul 24 14:01:41.059826 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.059811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:41.062441 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.062418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:01:41.062578 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.062561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:01:41.069229 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.069206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"]
Jul 24 14:01:41.151963 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.151928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr\" (UID: \"05723b48-7fb8-480a-889e-1c58b0daa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:41.252689 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.252647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr\" (UID: \"05723b48-7fb8-480a-889e-1c58b0daa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:41.253354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.253326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr\" (UID: \"05723b48-7fb8-480a-889e-1c58b0daa4af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:41.370988 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.370909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:41.507992 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.507968    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"]
Jul 24 14:01:41.510529 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:01:41.510498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05723b48_7fb8_480a_889e_1c58b0daa4af.slice/crio-f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e WatchSource:0}: Error finding container f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e: Status 404 returned error can't find the container with id f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e
Jul 24 14:01:41.512441 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.512426    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:01:41.530615 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:41.530591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr" event={"ID":"05723b48-7fb8-480a-889e-1c58b0daa4af","Type":"ContainerStarted","Data":"f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e"}
Jul 24 14:01:44.546401 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:44.546366    2579 generic.go:356] "Generic (PLEG): container finished" podID="05723b48-7fb8-480a-889e-1c58b0daa4af" containerID="3c122b35d67c2ddaf62ee01f1369703b1af2261c70acbbd81e9752d65ef886f6" exitCode=0
Jul 24 14:01:44.546791 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:44.546417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr" event={"ID":"05723b48-7fb8-480a-889e-1c58b0daa4af","Type":"ContainerDied","Data":"3c122b35d67c2ddaf62ee01f1369703b1af2261c70acbbd81e9752d65ef886f6"}
Jul 24 14:01:45.676856 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:45.676830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:45.794689 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:45.794597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca\") pod \"05723b48-7fb8-480a-889e-1c58b0daa4af\" (UID: \"05723b48-7fb8-480a-889e-1c58b0daa4af\") "
Jul 24 14:01:45.794898 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:45.794877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05723b48-7fb8-480a-889e-1c58b0daa4af" (UID: "05723b48-7fb8-480a-889e-1c58b0daa4af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:01:45.895847 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:45.895766    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05723b48-7fb8-480a-889e-1c58b0daa4af-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:01:46.556088 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:46.556060    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"
Jul 24 14:01:46.556088 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:46.556087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr" event={"ID":"05723b48-7fb8-480a-889e-1c58b0daa4af","Type":"ContainerDied","Data":"f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e"}
Jul 24 14:01:46.556300 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:46.556128    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88e0d199e86176d019361c7adb04ad5b531699a0d6b70f64a9dd3045289112e"
Jul 24 14:01:56.182664 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:56.182625    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"]
Jul 24 14:01:56.186111 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:56.186087    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4dc6ea7f-dqpwr"]
Jul 24 14:01:57.124334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:01:57.124298    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05723b48-7fb8-480a-889e-1c58b0daa4af" path="/var/lib/kubelet/pods/05723b48-7fb8-480a-889e-1c58b0daa4af/volumes"
Jul 24 14:02:25.888974 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.888928    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"]
Jul 24 14:02:25.889533 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.889460    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05723b48-7fb8-480a-889e-1c58b0daa4af" containerName="revoke-keys"
Jul 24 14:02:25.889533 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.889500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05723b48-7fb8-480a-889e-1c58b0daa4af" containerName="revoke-keys"
Jul 24 14:02:25.889659 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.889593    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05723b48-7fb8-480a-889e-1c58b0daa4af" containerName="revoke-keys"
Jul 24 14:02:25.891574 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.891553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:25.898095 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.897639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:02:25.898095 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.898055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:02:25.904690 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.904667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"]
Jul 24 14:02:25.957191 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:25.957146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v\" (UID: \"6e62a544-7d39-432c-b8e6-4871f0972bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:26.057667 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.057625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v\" (UID: \"6e62a544-7d39-432c-b8e6-4871f0972bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:26.058202 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.058184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v\" (UID: \"6e62a544-7d39-432c-b8e6-4871f0972bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:26.201763 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.201684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:26.336634 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.336459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"]
Jul 24 14:02:26.339214 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:02:26.339188    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e62a544_7d39_432c_b8e6_4871f0972bd3.slice/crio-3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6 WatchSource:0}: Error finding container 3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6: Status 404 returned error can't find the container with id 3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6
Jul 24 14:02:26.740219 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.740178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v" event={"ID":"6e62a544-7d39-432c-b8e6-4871f0972bd3","Type":"ContainerStarted","Data":"83a86028f3fec7cb4474efc290966d82c015eb74e04ce89694f74f2eed188140"}
Jul 24 14:02:26.740413 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.740233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v" event={"ID":"6e62a544-7d39-432c-b8e6-4871f0972bd3","Type":"ContainerStarted","Data":"3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6"}
Jul 24 14:02:26.761414 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:26.761365    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v" podStartSLOduration=1.761349696 podStartE2EDuration="1.761349696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:02:26.761271547 +0000 UTC m=+1686.223878394" watchObservedRunningTime="2026-07-24 14:02:26.761349696 +0000 UTC m=+1686.223956547"
Jul 24 14:02:27.745636 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:27.745601    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e62a544-7d39-432c-b8e6-4871f0972bd3" containerID="83a86028f3fec7cb4474efc290966d82c015eb74e04ce89694f74f2eed188140" exitCode=0
Jul 24 14:02:27.746109 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:27.745681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v" event={"ID":"6e62a544-7d39-432c-b8e6-4871f0972bd3","Type":"ContainerDied","Data":"83a86028f3fec7cb4474efc290966d82c015eb74e04ce89694f74f2eed188140"}
Jul 24 14:02:28.883443 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:28.883421    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:28.982806 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:28.982767    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca\") pod \"6e62a544-7d39-432c-b8e6-4871f0972bd3\" (UID: \"6e62a544-7d39-432c-b8e6-4871f0972bd3\") "
Jul 24 14:02:28.983150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:28.983129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e62a544-7d39-432c-b8e6-4871f0972bd3" (UID: "6e62a544-7d39-432c-b8e6-4871f0972bd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:02:29.083728 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:29.083683    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e62a544-7d39-432c-b8e6-4871f0972bd3-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:02:29.755167 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:29.755134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"
Jul 24 14:02:29.755357 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:29.755134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v" event={"ID":"6e62a544-7d39-432c-b8e6-4871f0972bd3","Type":"ContainerDied","Data":"3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6"}
Jul 24 14:02:29.755357 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:29.755271    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4c532c974457383752e5263af19f1087eb618ce7669930200ba6ccf142e8d6"
Jul 24 14:02:41.004805 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:41.004764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"]
Jul 24 14:02:41.006467 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:41.006444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4a9d360c-dvw8v"]
Jul 24 14:02:41.124334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:02:41.124262    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e62a544-7d39-432c-b8e6-4871f0972bd3" path="/var/lib/kubelet/pods/6e62a544-7d39-432c-b8e6-4871f0972bd3/volumes"
Jul 24 14:03:10.816156 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.816121    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"]
Jul 24 14:03:10.816650 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.816468    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e62a544-7d39-432c-b8e6-4871f0972bd3" containerName="revoke-keys"
Jul 24 14:03:10.816650 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.816493    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e62a544-7d39-432c-b8e6-4871f0972bd3" containerName="revoke-keys"
Jul 24 14:03:10.816650 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.816560    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e62a544-7d39-432c-b8e6-4871f0972bd3" containerName="revoke-keys"
Jul 24 14:03:10.818721 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.818706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:10.823023 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.822995    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:03:10.823156 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.823057    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:03:10.829781 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.829751    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"]
Jul 24 14:03:10.849372 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.849341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx\" (UID: \"e499c3a8-96a6-48a6-9ddd-f739b561d163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:10.950191 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.950153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx\" (UID: \"e499c3a8-96a6-48a6-9ddd-f739b561d163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:10.950865 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:10.950844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx\" (UID: \"e499c3a8-96a6-48a6-9ddd-f739b561d163\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:11.128762 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:11.128688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:11.256079 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:11.256053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"]
Jul 24 14:03:11.258749 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:03:11.258723    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode499c3a8_96a6_48a6_9ddd_f739b561d163.slice/crio-cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a WatchSource:0}: Error finding container cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a: Status 404 returned error can't find the container with id cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a
Jul 24 14:03:11.944734 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:11.944696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx" event={"ID":"e499c3a8-96a6-48a6-9ddd-f739b561d163","Type":"ContainerStarted","Data":"1e9659a97ee6be03e3dd92ab266112790e861c0a09e94056e4b2dddee0be0ddd"}
Jul 24 14:03:11.945117 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:11.944782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx" event={"ID":"e499c3a8-96a6-48a6-9ddd-f739b561d163","Type":"ContainerStarted","Data":"cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a"}
Jul 24 14:03:11.964540 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:11.964489    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx" podStartSLOduration=1.964457017 podStartE2EDuration="1.964457017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:03:11.961890171 +0000 UTC m=+1731.424497015" watchObservedRunningTime="2026-07-24 14:03:11.964457017 +0000 UTC m=+1731.427063864"
Jul 24 14:03:12.950246 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:12.950209    2579 generic.go:356] "Generic (PLEG): container finished" podID="e499c3a8-96a6-48a6-9ddd-f739b561d163" containerID="1e9659a97ee6be03e3dd92ab266112790e861c0a09e94056e4b2dddee0be0ddd" exitCode=0
Jul 24 14:03:12.950653 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:12.950270    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx" event={"ID":"e499c3a8-96a6-48a6-9ddd-f739b561d163","Type":"ContainerDied","Data":"1e9659a97ee6be03e3dd92ab266112790e861c0a09e94056e4b2dddee0be0ddd"}
Jul 24 14:03:14.081385 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.081363    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:14.176456 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.176413    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca\") pod \"e499c3a8-96a6-48a6-9ddd-f739b561d163\" (UID: \"e499c3a8-96a6-48a6-9ddd-f739b561d163\") "
Jul 24 14:03:14.176816 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.176794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e499c3a8-96a6-48a6-9ddd-f739b561d163" (UID: "e499c3a8-96a6-48a6-9ddd-f739b561d163"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:03:14.277455 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.277422    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e499c3a8-96a6-48a6-9ddd-f739b561d163-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:03:14.960458 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.960426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"
Jul 24 14:03:14.960644 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.960430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx" event={"ID":"e499c3a8-96a6-48a6-9ddd-f739b561d163","Type":"ContainerDied","Data":"cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a"}
Jul 24 14:03:14.960644 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:14.960535    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfa985d642aacb420d509d5fa4ba90ccdc0b393ecf59b8054ba066fe2d35230a"
Jul 24 14:03:25.926456 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:25.926423    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"]
Jul 24 14:03:25.930019 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:25.929982    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6538ff29-ptzwx"]
Jul 24 14:03:27.124758 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:27.124723    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e499c3a8-96a6-48a6-9ddd-f739b561d163" path="/var/lib/kubelet/pods/e499c3a8-96a6-48a6-9ddd-f739b561d163/volumes"
Jul 24 14:03:55.249415 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.249378    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"]
Jul 24 14:03:55.249906 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.249757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e499c3a8-96a6-48a6-9ddd-f739b561d163" containerName="revoke-keys"
Jul 24 14:03:55.249906 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.249769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e499c3a8-96a6-48a6-9ddd-f739b561d163" containerName="revoke-keys"
Jul 24 14:03:55.249906 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.249817    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e499c3a8-96a6-48a6-9ddd-f739b561d163" containerName="revoke-keys"
Jul 24 14:03:55.252701 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.252685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:55.255531 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.255508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:03:55.255830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.255809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:03:55.264701 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.264676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"]
Jul 24 14:03:55.345590 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.345555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9846c499-lf9vc\" (UID: \"92cfc925-056b-4f16-8625-ca9ebf2de41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:55.446561 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.446522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9846c499-lf9vc\" (UID: \"92cfc925-056b-4f16-8625-ca9ebf2de41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:55.447126 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.447105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9846c499-lf9vc\" (UID: \"92cfc925-056b-4f16-8625-ca9ebf2de41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:55.562589 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.562546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:55.714859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:55.714829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"]
Jul 24 14:03:55.718642 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:03:55.718614    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cfc925_056b_4f16_8625_ca9ebf2de41c.slice/crio-e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386 WatchSource:0}: Error finding container e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386: Status 404 returned error can't find the container with id e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386
Jul 24 14:03:56.151317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:56.151219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc" event={"ID":"92cfc925-056b-4f16-8625-ca9ebf2de41c","Type":"ContainerStarted","Data":"7ac3fe6d0bb00ee76e07a95418603baeb8d2c32ec105c8e5d72006504580191a"}
Jul 24 14:03:56.151317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:56.151288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc" event={"ID":"92cfc925-056b-4f16-8625-ca9ebf2de41c","Type":"ContainerStarted","Data":"e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386"}
Jul 24 14:03:56.193649 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:56.193600    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc" podStartSLOduration=1.193584044 podStartE2EDuration="1.193584044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:03:56.193382492 +0000 UTC m=+1775.655989367" watchObservedRunningTime="2026-07-24 14:03:56.193584044 +0000 UTC m=+1775.656190887"
Jul 24 14:03:57.157655 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:57.157622    2579 generic.go:356] "Generic (PLEG): container finished" podID="92cfc925-056b-4f16-8625-ca9ebf2de41c" containerID="7ac3fe6d0bb00ee76e07a95418603baeb8d2c32ec105c8e5d72006504580191a" exitCode=0
Jul 24 14:03:57.158066 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:57.157699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc" event={"ID":"92cfc925-056b-4f16-8625-ca9ebf2de41c","Type":"ContainerDied","Data":"7ac3fe6d0bb00ee76e07a95418603baeb8d2c32ec105c8e5d72006504580191a"}
Jul 24 14:03:58.289168 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:58.289144    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:58.368854 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:58.368828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca\") pod \"92cfc925-056b-4f16-8625-ca9ebf2de41c\" (UID: \"92cfc925-056b-4f16-8625-ca9ebf2de41c\") "
Jul 24 14:03:58.369151 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:58.369128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92cfc925-056b-4f16-8625-ca9ebf2de41c" (UID: "92cfc925-056b-4f16-8625-ca9ebf2de41c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:03:58.470034 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:58.469946    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cfc925-056b-4f16-8625-ca9ebf2de41c-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:03:59.171418 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:59.171373    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"
Jul 24 14:03:59.171418 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:59.171406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc" event={"ID":"92cfc925-056b-4f16-8625-ca9ebf2de41c","Type":"ContainerDied","Data":"e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386"}
Jul 24 14:03:59.171625 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:03:59.171455    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9705e352dc3835dd25e1df720817eccd411b2c86ab2878f0df9bbbc95823386"
Jul 24 14:04:10.343363 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:04:10.343313    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"]
Jul 24 14:04:10.348796 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:04:10.348764    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9846c499-lf9vc"]
Jul 24 14:04:11.124067 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:04:11.123989    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cfc925-056b-4f16-8625-ca9ebf2de41c" path="/var/lib/kubelet/pods/92cfc925-056b-4f16-8625-ca9ebf2de41c/volumes"
Jul 24 14:04:21.080109 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:04:21.079991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:04:21.087982 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:04:21.085459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:06:18.387395 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.387353    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"]
Jul 24 14:06:18.387877 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.387746    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cfc925-056b-4f16-8625-ca9ebf2de41c" containerName="revoke-keys"
Jul 24 14:06:18.387877 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.387762    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cfc925-056b-4f16-8625-ca9ebf2de41c" containerName="revoke-keys"
Jul 24 14:06:18.387877 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.387820    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cfc925-056b-4f16-8625-ca9ebf2de41c" containerName="revoke-keys"
Jul 24 14:06:18.390671 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.390656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:18.394740 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.394714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:06:18.395317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.395289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:06:18.406184 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.406116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"]
Jul 24 14:06:18.512046 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.512007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-296b2828-wppcx\" (UID: \"bc3786f5-8187-4606-96e7-ccabc59f6dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:18.612812 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.612773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-296b2828-wppcx\" (UID: \"bc3786f5-8187-4606-96e7-ccabc59f6dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:18.613400 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.613379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-296b2828-wppcx\" (UID: \"bc3786f5-8187-4606-96e7-ccabc59f6dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:18.702021 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.701927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:18.844037 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:18.844012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"]
Jul 24 14:06:18.846180 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:06:18.846152    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3786f5_8187_4606_96e7_ccabc59f6dd0.slice/crio-688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c WatchSource:0}: Error finding container 688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c: Status 404 returned error can't find the container with id 688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c
Jul 24 14:06:19.827814 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:19.827768    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc3786f5-8187-4606-96e7-ccabc59f6dd0" containerID="6a9a4a395248b7be5a5f8476a756d70a95e7a14d8170c068c25394797dbebb08" exitCode=0
Jul 24 14:06:19.827814 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:19.827806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx" event={"ID":"bc3786f5-8187-4606-96e7-ccabc59f6dd0","Type":"ContainerDied","Data":"6a9a4a395248b7be5a5f8476a756d70a95e7a14d8170c068c25394797dbebb08"}
Jul 24 14:06:19.828288 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:19.827852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx" event={"ID":"bc3786f5-8187-4606-96e7-ccabc59f6dd0","Type":"ContainerStarted","Data":"688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c"}
Jul 24 14:06:20.973019 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:20.972994    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:21.133487 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.133397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca\") pod \"bc3786f5-8187-4606-96e7-ccabc59f6dd0\" (UID: \"bc3786f5-8187-4606-96e7-ccabc59f6dd0\") "
Jul 24 14:06:21.133737 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.133716    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc3786f5-8187-4606-96e7-ccabc59f6dd0" (UID: "bc3786f5-8187-4606-96e7-ccabc59f6dd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:06:21.234254 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.234214    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3786f5-8187-4606-96e7-ccabc59f6dd0-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:06:21.840195 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.840164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"
Jul 24 14:06:21.840195 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.840168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx" event={"ID":"bc3786f5-8187-4606-96e7-ccabc59f6dd0","Type":"ContainerDied","Data":"688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c"}
Jul 24 14:06:21.840440 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:21.840217    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="688790755c30356548d2f93898389e632ec3f319a05c8c48d82d1928c81f7f6c"
Jul 24 14:06:33.474452 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:33.474413    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"]
Jul 24 14:06:33.477622 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:33.477588    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-296b2828-wppcx"]
Jul 24 14:06:35.129446 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:06:35.129408    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3786f5-8187-4606-96e7-ccabc59f6dd0" path="/var/lib/kubelet/pods/bc3786f5-8187-4606-96e7-ccabc59f6dd0/volumes"
Jul 24 14:07:39.554608 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.554572    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:07:39.555061 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.554922    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3786f5-8187-4606-96e7-ccabc59f6dd0" containerName="revoke-keys"
Jul 24 14:07:39.555061 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.554933    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3786f5-8187-4606-96e7-ccabc59f6dd0" containerName="revoke-keys"
Jul 24 14:07:39.555061 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.554987    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3786f5-8187-4606-96e7-ccabc59f6dd0" containerName="revoke-keys"
Jul 24 14:07:39.557889 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.557874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.564876 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.564853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3098bd50\"/\"kube-root-ca.crt\""
Jul 24 14:07:39.565846 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.565829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3098bd50\"/\"e2e-lifecycle-model-3098bd50-kserve-self-signed-certs\""
Jul 24 14:07:39.567910 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.567893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3098bd50\"/\"openshift-service-ca.crt\""
Jul 24 14:07:39.567910 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.567906    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3098bd50\"/\"default-dockercfg-kf865\""
Jul 24 14:07:39.589790 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.589759    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:07:39.679440 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.679643 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.679643 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.679643 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzxp\" (UniqueName: \"kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.679772 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.679772 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.679679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780231 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780231 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzxp\" (UniqueName: \"kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780785 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780855 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.780855 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.780814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.782651 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.782633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.783003 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.782988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.790566 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.790537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzxp\" (UniqueName: \"kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp\") pod \"e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") " pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:39.866947 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:39.866847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:40.015572 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:40.015537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:07:40.019255 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:07:40.019228    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e7929a_0fc2_41b4_a302_4d1273b5094f.slice/crio-678f8a0edfd3758c32aa9bb346fef07abc42af8fdb86a11f396679feaddfcf28 WatchSource:0}: Error finding container 678f8a0edfd3758c32aa9bb346fef07abc42af8fdb86a11f396679feaddfcf28: Status 404 returned error can't find the container with id 678f8a0edfd3758c32aa9bb346fef07abc42af8fdb86a11f396679feaddfcf28
Jul 24 14:07:40.201144 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:40.201042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" event={"ID":"91e7929a-0fc2-41b4-a302-4d1273b5094f","Type":"ContainerStarted","Data":"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"}
Jul 24 14:07:40.201144 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:40.201096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" event={"ID":"91e7929a-0fc2-41b4-a302-4d1273b5094f","Type":"ContainerStarted","Data":"678f8a0edfd3758c32aa9bb346fef07abc42af8fdb86a11f396679feaddfcf28"}
Jul 24 14:07:40.256187 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:40.256131    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" podStartSLOduration=1.256110558 podStartE2EDuration="1.256110558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:40.255003584 +0000 UTC m=+1999.717610428" watchObservedRunningTime="2026-07-24 14:07:40.256110558 +0000 UTC m=+1999.718717402"
Jul 24 14:07:49.867460 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:49.867416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:49.868078 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:49.867513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:49.880109 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:49.880081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:07:50.259447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:07:50.259419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:08:04.944829 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:04.944790    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"]
Jul 24 14:08:04.948322 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:04.948302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:04.951366 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:04.951345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:08:04.951676 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:04.951658    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:08:04.970317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:04.970296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"]
Jul 24 14:08:05.106706 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:05.106669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m\" (UID: \"2dd54c11-1155-4d50-a011-d9e1cef4a36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:05.208010 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:05.207909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m\" (UID: \"2dd54c11-1155-4d50-a011-d9e1cef4a36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:05.208737 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:05.208713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m\" (UID: \"2dd54c11-1155-4d50-a011-d9e1cef4a36b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:05.259124 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:05.259082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:05.402808 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:05.402778    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"]
Jul 24 14:08:05.415460 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:08:05.415435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd54c11_1155_4d50_a011_d9e1cef4a36b.slice/crio-dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b WatchSource:0}: Error finding container dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b: Status 404 returned error can't find the container with id dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b
Jul 24 14:08:06.326030 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:06.325995    2579 generic.go:356] "Generic (PLEG): container finished" podID="2dd54c11-1155-4d50-a011-d9e1cef4a36b" containerID="4dc72c7cf923cf48eb591ccf890b7fe133795a8739e022b3fa50bfd2a6311055" exitCode=0
Jul 24 14:08:06.326402 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:06.326039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m" event={"ID":"2dd54c11-1155-4d50-a011-d9e1cef4a36b","Type":"ContainerDied","Data":"4dc72c7cf923cf48eb591ccf890b7fe133795a8739e022b3fa50bfd2a6311055"}
Jul 24 14:08:06.326402 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:06.326089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m" event={"ID":"2dd54c11-1155-4d50-a011-d9e1cef4a36b","Type":"ContainerStarted","Data":"dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b"}
Jul 24 14:08:07.464259 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:07.464229    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:07.631775 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:07.631680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca\") pod \"2dd54c11-1155-4d50-a011-d9e1cef4a36b\" (UID: \"2dd54c11-1155-4d50-a011-d9e1cef4a36b\") "
Jul 24 14:08:07.632079 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:07.632057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2dd54c11-1155-4d50-a011-d9e1cef4a36b" (UID: "2dd54c11-1155-4d50-a011-d9e1cef4a36b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:08:07.732499 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:07.732443    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd54c11-1155-4d50-a011-d9e1cef4a36b-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:08.337272 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:08.337240    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"
Jul 24 14:08:08.337272 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:08.337263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m" event={"ID":"2dd54c11-1155-4d50-a011-d9e1cef4a36b","Type":"ContainerDied","Data":"dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b"}
Jul 24 14:08:08.337472 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:08.337307    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc38ea05bc28f8c5ef8fadc2ed1c802c82f37fc2f23d4d4b2a8e9c168489d07b"
Jul 24 14:08:21.178462 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:21.178428    2579 scope.go:117] "RemoveContainer" containerID="3c122b35d67c2ddaf62ee01f1369703b1af2261c70acbbd81e9752d65ef886f6"
Jul 24 14:08:25.098407 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:25.098371    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"]
Jul 24 14:08:25.107382 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:25.107349    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3098bd50-fcs2m"]
Jul 24 14:08:25.124541 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:25.124509    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd54c11-1155-4d50-a011-d9e1cef4a36b" path="/var/lib/kubelet/pods/2dd54c11-1155-4d50-a011-d9e1cef4a36b/volumes"
Jul 24 14:08:33.674328 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:33.674291    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:08:48.693968 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:48.693852    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" podUID="91e7929a-0fc2-41b4-a302-4d1273b5094f" containerName="main" containerID="cri-o://07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7" gracePeriod=45
Jul 24 14:08:48.935697 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:48.935674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:08:49.094385 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwzxp\" (UniqueName: \"kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home\") pod \"91e7929a-0fc2-41b4-a302-4d1273b5094f\" (UID: \"91e7929a-0fc2-41b4-a302-4d1273b5094f\") "
Jul 24 14:08:49.094931 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:08:49.094931 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache" (OuterVolumeSpecName: "model-cache") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:08:49.094931 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.094908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home" (OuterVolumeSpecName: "home") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:08:49.096964 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.096925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm" (OuterVolumeSpecName: "dshm") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:08:49.097073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.096995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp" (OuterVolumeSpecName: "kube-api-access-xwzxp") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "kube-api-access-xwzxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:08:49.097073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.097000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91e7929a-0fc2-41b4-a302-4d1273b5094f" (UID: "91e7929a-0fc2-41b4-a302-4d1273b5094f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:08:49.195742 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195697    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzxp\" (UniqueName: \"kubernetes.io/projected/91e7929a-0fc2-41b4-a302-4d1273b5094f-kube-api-access-xwzxp\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.195742 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195739    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.195742 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195752    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.195997 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195762    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.195997 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195770    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e7929a-0fc2-41b4-a302-4d1273b5094f-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.195997 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.195776    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e7929a-0fc2-41b4-a302-4d1273b5094f-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:08:49.527258 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.527223    2579 generic.go:356] "Generic (PLEG): container finished" podID="91e7929a-0fc2-41b4-a302-4d1273b5094f" containerID="07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7" exitCode=0
Jul 24 14:08:49.527428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.527265    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"
Jul 24 14:08:49.527428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.527293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" event={"ID":"91e7929a-0fc2-41b4-a302-4d1273b5094f","Type":"ContainerDied","Data":"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"}
Jul 24 14:08:49.527428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.527328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t" event={"ID":"91e7929a-0fc2-41b4-a302-4d1273b5094f","Type":"ContainerDied","Data":"678f8a0edfd3758c32aa9bb346fef07abc42af8fdb86a11f396679feaddfcf28"}
Jul 24 14:08:49.527428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.527348    2579 scope.go:117] "RemoveContainer" containerID="07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"
Jul 24 14:08:49.536702 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.536679    2579 scope.go:117] "RemoveContainer" containerID="07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"
Jul 24 14:08:49.536958 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:08:49.536938    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7\": container with ID starting with 07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7 not found: ID does not exist" containerID="07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"
Jul 24 14:08:49.537025 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.536968    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7"} err="failed to get container status \"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7\": rpc error: code = NotFound desc = could not find container \"07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7\": container with ID starting with 07d02a2bc6933e4ef9411ff98b7cdd56c3b64284c64d5e5721101c0027f41dd7 not found: ID does not exist"
Jul 24 14:08:49.547180 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.547146    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:08:49.549050 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:49.549027    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3098bd50/e2e-lifecycle-model-3098bd50-kserve-5999bbcfcf-pkc8t"]
Jul 24 14:08:51.125269 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:08:51.125222    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e7929a-0fc2-41b4-a302-4d1273b5094f" path="/var/lib/kubelet/pods/91e7929a-0fc2-41b4-a302-4d1273b5094f/volumes"
Jul 24 14:09:21.108061 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:09:21.107948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:09:21.113966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:09:21.113948    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:09:21.197856 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:09:21.197826    2579 scope.go:117] "RemoveContainer" containerID="1e9659a97ee6be03e3dd92ab266112790e861c0a09e94056e4b2dddee0be0ddd"
Jul 24 14:09:21.205829 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:09:21.205803    2579 scope.go:117] "RemoveContainer" containerID="83a86028f3fec7cb4474efc290966d82c015eb74e04ce89694f74f2eed188140"
Jul 24 14:10:21.231403 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:21.231368    2579 scope.go:117] "RemoveContainer" containerID="7ac3fe6d0bb00ee76e07a95418603baeb8d2c32ec105c8e5d72006504580191a"
Jul 24 14:10:44.612808 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.612769    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd54c11-1155-4d50-a011-d9e1cef4a36b" containerName="revoke-keys"
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613101    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd54c11-1155-4d50-a011-d9e1cef4a36b" containerName="revoke-keys"
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613123    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e7929a-0fc2-41b4-a302-4d1273b5094f" containerName="main"
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613129    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e7929a-0fc2-41b4-a302-4d1273b5094f" containerName="main"
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613182    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd54c11-1155-4d50-a011-d9e1cef4a36b" containerName="revoke-keys"
Jul 24 14:10:44.613275 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.613190    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e7929a-0fc2-41b4-a302-4d1273b5094f" containerName="main"
Jul 24 14:10:44.616169 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.616150    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.620354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.620328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c21e7d\"/\"default-dockercfg-5bfsl\""
Jul 24 14:10:44.620521 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.620363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c21e7d\"/\"openshift-service-ca.crt\""
Jul 24 14:10:44.620521 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.620370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c21e7d\"/\"kube-root-ca.crt\""
Jul 24 14:10:44.620780 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.620763    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c21e7d\"/\"test-model-c21e7d-kserve-self-signed-certs\""
Jul 24 14:10:44.632459 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.632416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:10:44.710687 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.710873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.710873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.710873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rww2\" (UniqueName: \"kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.710873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.711012 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.710885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811538 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rww2\" (UniqueName: \"kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.811753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.812034 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.811973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.812034 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.812020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.812133 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.812045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.813908 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.813884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.814150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.814134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.820764 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.820743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rww2\" (UniqueName: \"kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2\") pod \"test-model-c21e7d-kserve-558885c657-qswk9\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") " pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:44.929208 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:44.929121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:45.066672 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:45.066453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:10:45.069167 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:10:45.069139    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8fb2761_89af_4792_a5a3_695da5903c40.slice/crio-0d494b97a46e51caf43c994a4c8bb6055fcfa4854c68df53f4f70bcb593b90be WatchSource:0}: Error finding container 0d494b97a46e51caf43c994a4c8bb6055fcfa4854c68df53f4f70bcb593b90be: Status 404 returned error can't find the container with id 0d494b97a46e51caf43c994a4c8bb6055fcfa4854c68df53f4f70bcb593b90be
Jul 24 14:10:46.074270 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:46.074230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" event={"ID":"e8fb2761-89af-4792-a5a3-695da5903c40","Type":"ContainerStarted","Data":"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"}
Jul 24 14:10:46.074694 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:46.074279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" event={"ID":"e8fb2761-89af-4792-a5a3-695da5903c40","Type":"ContainerStarted","Data":"0d494b97a46e51caf43c994a4c8bb6055fcfa4854c68df53f4f70bcb593b90be"}
Jul 24 14:10:46.103232 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:46.103180    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" podStartSLOduration=2.103165149 podStartE2EDuration="2.103165149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:10:46.101419923 +0000 UTC m=+2185.564026767" watchObservedRunningTime="2026-07-24 14:10:46.103165149 +0000 UTC m=+2185.565772005"
Jul 24 14:10:54.930280 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:54.930236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:54.930781 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:54.930294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:54.942658 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:54.942639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:10:55.127931 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:10:55.127899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:11:10.705831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.705799    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:11:10.709108 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.709084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.712068 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.712041    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d52a8f\"/\"kube-root-ca.crt\""
Jul 24 14:11:10.713150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.713123    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d52a8f\"/\"test-model-d52a8f-kserve-self-signed-certs\""
Jul 24 14:11:10.713261 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.713126    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d52a8f\"/\"openshift-service-ca.crt\""
Jul 24 14:11:10.713261 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.713131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d52a8f\"/\"default-dockercfg-2psr8\""
Jul 24 14:11:10.720948 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.720923    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:11:10.848195 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.848377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.848377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.848377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27dt6\" (UniqueName: \"kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.848377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.848545 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.848420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949611 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949611 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27dt6\" (UniqueName: \"kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.949859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.950067 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.949999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.950106 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.950072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.950106 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.950090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.952020 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.952000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.952289 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.952271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:10.958579 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:10.958510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27dt6\" (UniqueName: \"kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6\") pod \"test-model-d52a8f-kserve-bf7d89985-5zfbn\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") " pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:11.020353 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:11.020311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:11.156627 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:11.156602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:11:11.159120 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:11:11.159084    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b050b70_7202_4410_957a_94821adb7226.slice/crio-c3155ba41ffc38aaa2d4ccb2407442a089c90272b5397a74d4eeecd2bd3a5542 WatchSource:0}: Error finding container c3155ba41ffc38aaa2d4ccb2407442a089c90272b5397a74d4eeecd2bd3a5542: Status 404 returned error can't find the container with id c3155ba41ffc38aaa2d4ccb2407442a089c90272b5397a74d4eeecd2bd3a5542
Jul 24 14:11:11.192387 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:11.192341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" event={"ID":"4b050b70-7202-4410-957a-94821adb7226","Type":"ContainerStarted","Data":"c3155ba41ffc38aaa2d4ccb2407442a089c90272b5397a74d4eeecd2bd3a5542"}
Jul 24 14:11:12.200130 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:12.200090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" event={"ID":"4b050b70-7202-4410-957a-94821adb7226","Type":"ContainerStarted","Data":"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"}
Jul 24 14:11:12.226544 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:12.226488    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" podStartSLOduration=2.226457899 podStartE2EDuration="2.226457899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:12.22373851 +0000 UTC m=+2211.686345356" watchObservedRunningTime="2026-07-24 14:11:12.226457899 +0000 UTC m=+2211.689064744"
Jul 24 14:11:21.020993 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:21.020939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:21.020993 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:21.021001    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:21.034006 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:21.033975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:21.251321 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:21.251293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:11:50.399550 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:50.399455    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:11:51.417868 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.417834    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:11:51.557341 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.557305    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"]
Jul 24 14:11:51.560890 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.560875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:51.563320 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.563299    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:11:51.563438 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.563354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:11:51.569669 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.569646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"]
Jul 24 14:11:51.713926 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.713837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm\" (UID: \"78a03ebb-5868-4834-8100-6fcaac63d413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:51.814732 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.814690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm\" (UID: \"78a03ebb-5868-4834-8100-6fcaac63d413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:51.815414 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.815393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm\" (UID: \"78a03ebb-5868-4834-8100-6fcaac63d413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:51.872830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:51.872795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:52.008833 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:52.008809    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"]
Jul 24 14:11:52.011421 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:11:52.011392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a03ebb_5868_4834_8100_6fcaac63d413.slice/crio-ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53 WatchSource:0}: Error finding container ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53: Status 404 returned error can't find the container with id ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53
Jul 24 14:11:52.391744 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:52.391710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm" event={"ID":"78a03ebb-5868-4834-8100-6fcaac63d413","Type":"ContainerStarted","Data":"2f03ea6b1c3a32cb53b3ada2df79b53d0eb5416f53e6f7e8fe8bf2215f68af5b"}
Jul 24 14:11:52.391958 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:52.391785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm" event={"ID":"78a03ebb-5868-4834-8100-6fcaac63d413","Type":"ContainerStarted","Data":"ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53"}
Jul 24 14:11:52.409068 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:52.409020    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm" podStartSLOduration=1.409005306 podStartE2EDuration="1.409005306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:11:52.408170794 +0000 UTC m=+2251.870777639" watchObservedRunningTime="2026-07-24 14:11:52.409005306 +0000 UTC m=+2251.871612156"
Jul 24 14:11:53.398063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:53.398024    2579 generic.go:356] "Generic (PLEG): container finished" podID="78a03ebb-5868-4834-8100-6fcaac63d413" containerID="2f03ea6b1c3a32cb53b3ada2df79b53d0eb5416f53e6f7e8fe8bf2215f68af5b" exitCode=0
Jul 24 14:11:53.398547 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:53.398111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm" event={"ID":"78a03ebb-5868-4834-8100-6fcaac63d413","Type":"ContainerDied","Data":"2f03ea6b1c3a32cb53b3ada2df79b53d0eb5416f53e6f7e8fe8bf2215f68af5b"}
Jul 24 14:11:54.531736 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:54.531710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:54.640493 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:54.640454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca\") pod \"78a03ebb-5868-4834-8100-6fcaac63d413\" (UID: \"78a03ebb-5868-4834-8100-6fcaac63d413\") "
Jul 24 14:11:54.640860 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:54.640834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78a03ebb-5868-4834-8100-6fcaac63d413" (UID: "78a03ebb-5868-4834-8100-6fcaac63d413"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:11:54.741184 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:54.741088    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78a03ebb-5868-4834-8100-6fcaac63d413-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:11:55.408695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:55.408607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm" event={"ID":"78a03ebb-5868-4834-8100-6fcaac63d413","Type":"ContainerDied","Data":"ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53"}
Jul 24 14:11:55.408695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:55.408635    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"
Jul 24 14:11:55.408695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:11:55.408670    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef062003059d9667fd9e6723953556e7fdd704fce1bf47f8b494705545129e53"
Jul 24 14:12:05.418967 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.418895    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" podUID="e8fb2761-89af-4792-a5a3-695da5903c40" containerName="main" containerID="cri-o://47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89" gracePeriod=45
Jul 24 14:12:05.666233 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.666208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:12:05.839236 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839435 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839435 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839278    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839435 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839298    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839435 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839435 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rww2\" (UniqueName: \"kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2\") pod \"e8fb2761-89af-4792-a5a3-695da5903c40\" (UID: \"e8fb2761-89af-4792-a5a3-695da5903c40\") "
Jul 24 14:12:05.839720 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache" (OuterVolumeSpecName: "model-cache") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:05.839720 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839646    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:05.839720 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home" (OuterVolumeSpecName: "home") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:05.839861 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839776    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:05.839861 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839796    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:05.839861 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.839810    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:05.841782 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.841756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm" (OuterVolumeSpecName: "dshm") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:05.841870 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.841799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:12:05.841916 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.841896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2" (OuterVolumeSpecName: "kube-api-access-9rww2") pod "e8fb2761-89af-4792-a5a3-695da5903c40" (UID: "e8fb2761-89af-4792-a5a3-695da5903c40"). InnerVolumeSpecName "kube-api-access-9rww2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:05.940942 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.940900    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8fb2761-89af-4792-a5a3-695da5903c40-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:05.940942 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.940935    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8fb2761-89af-4792-a5a3-695da5903c40-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:05.940942 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:05.940946    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rww2\" (UniqueName: \"kubernetes.io/projected/e8fb2761-89af-4792-a5a3-695da5903c40-kube-api-access-9rww2\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.437384 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.437324    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" podUID="4b050b70-7202-4410-957a-94821adb7226" containerName="main" containerID="cri-o://2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf" gracePeriod=45
Jul 24 14:12:06.468068 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.468038    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8fb2761-89af-4792-a5a3-695da5903c40" containerID="47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89" exitCode=0
Jul 24 14:12:06.468226 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.468083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"
Jul 24 14:12:06.468226 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.468120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" event={"ID":"e8fb2761-89af-4792-a5a3-695da5903c40","Type":"ContainerDied","Data":"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"}
Jul 24 14:12:06.468226 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.468168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9" event={"ID":"e8fb2761-89af-4792-a5a3-695da5903c40","Type":"ContainerDied","Data":"0d494b97a46e51caf43c994a4c8bb6055fcfa4854c68df53f4f70bcb593b90be"}
Jul 24 14:12:06.468226 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.468190    2579 scope.go:117] "RemoveContainer" containerID="47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"
Jul 24 14:12:06.523824 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.523795    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:12:06.526343 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.526319    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c21e7d/test-model-c21e7d-kserve-558885c657-qswk9"]
Jul 24 14:12:06.549011 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.548990    2579 scope.go:117] "RemoveContainer" containerID="47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"
Jul 24 14:12:06.549350 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:12:06.549319    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89\": container with ID starting with 47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89 not found: ID does not exist" containerID="47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"
Jul 24 14:12:06.549439 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.549358    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89"} err="failed to get container status \"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89\": rpc error: code = NotFound desc = could not find container \"47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89\": container with ID starting with 47e2d6e5f0f4c7faf0ec7c146440fac3148d4423b906375d0ddbab8243af6c89 not found: ID does not exist"
Jul 24 14:12:06.681302 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.681280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:12:06.702408 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.702339    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"]
Jul 24 14:12:06.706453 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.706428    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c21e7d-g5rxm"]
Jul 24 14:12:06.849886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.849851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27dt6\" (UniqueName: \"kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.849931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.849963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.849988    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.850024    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.850051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home\") pod \"4b050b70-7202-4410-957a-94821adb7226\" (UID: \"4b050b70-7202-4410-957a-94821adb7226\") "
Jul 24 14:12:06.850344 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.850282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:06.850344 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.850295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:06.850344 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.850322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home" (OuterVolumeSpecName: "home") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:06.852338 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.852311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm" (OuterVolumeSpecName: "dshm") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:12:06.852447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.852355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6" (OuterVolumeSpecName: "kube-api-access-27dt6") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "kube-api-access-27dt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:12:06.852447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.852394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b050b70-7202-4410-957a-94821adb7226" (UID: "4b050b70-7202-4410-957a-94821adb7226"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:12:06.951590 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951549    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.951590 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951579    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.951590 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951590    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b050b70-7202-4410-957a-94821adb7226-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.951590 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951598    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.951868 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951607    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27dt6\" (UniqueName: \"kubernetes.io/projected/4b050b70-7202-4410-957a-94821adb7226-kube-api-access-27dt6\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:06.951868 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:06.951616    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b050b70-7202-4410-957a-94821adb7226-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:07.125670 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.125634    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a03ebb-5868-4834-8100-6fcaac63d413" path="/var/lib/kubelet/pods/78a03ebb-5868-4834-8100-6fcaac63d413/volumes"
Jul 24 14:12:07.126016 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.126003    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fb2761-89af-4792-a5a3-695da5903c40" path="/var/lib/kubelet/pods/e8fb2761-89af-4792-a5a3-695da5903c40/volumes"
Jul 24 14:12:07.474427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.474342    2579 generic.go:356] "Generic (PLEG): container finished" podID="4b050b70-7202-4410-957a-94821adb7226" containerID="2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf" exitCode=0
Jul 24 14:12:07.474427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.474387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"
Jul 24 14:12:07.474427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.474412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" event={"ID":"4b050b70-7202-4410-957a-94821adb7226","Type":"ContainerDied","Data":"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"}
Jul 24 14:12:07.474963 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.474441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn" event={"ID":"4b050b70-7202-4410-957a-94821adb7226","Type":"ContainerDied","Data":"c3155ba41ffc38aaa2d4ccb2407442a089c90272b5397a74d4eeecd2bd3a5542"}
Jul 24 14:12:07.474963 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.474457    2579 scope.go:117] "RemoveContainer" containerID="2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"
Jul 24 14:12:07.482754 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.482732    2579 scope.go:117] "RemoveContainer" containerID="2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"
Jul 24 14:12:07.483030 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:12:07.483012    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf\": container with ID starting with 2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf not found: ID does not exist" containerID="2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"
Jul 24 14:12:07.483120 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.483038    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf"} err="failed to get container status \"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf\": rpc error: code = NotFound desc = could not find container \"2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf\": container with ID starting with 2f882f8aba050a6aefd090415f36cfe7d44b7dadf2c9d8ac96fba9bf070657bf not found: ID does not exist"
Jul 24 14:12:07.496189 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.496169    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:12:07.500500 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:07.500467    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d52a8f/test-model-d52a8f-kserve-bf7d89985-5zfbn"]
Jul 24 14:12:09.124161 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:09.124123    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b050b70-7202-4410-957a-94821adb7226" path="/var/lib/kubelet/pods/4b050b70-7202-4410-957a-94821adb7226/volumes"
Jul 24 14:12:14.092563 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092525    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"]
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092855    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fb2761-89af-4792-a5a3-695da5903c40" containerName="main"
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092865    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fb2761-89af-4792-a5a3-695da5903c40" containerName="main"
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b050b70-7202-4410-957a-94821adb7226" containerName="main"
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092887    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b050b70-7202-4410-957a-94821adb7226" containerName="main"
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092893    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a03ebb-5868-4834-8100-6fcaac63d413" containerName="revoke-keys"
Jul 24 14:12:14.092945 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092899    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a03ebb-5868-4834-8100-6fcaac63d413" containerName="revoke-keys"
Jul 24 14:12:14.093139 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092973    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b050b70-7202-4410-957a-94821adb7226" containerName="main"
Jul 24 14:12:14.093139 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092980    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a03ebb-5868-4834-8100-6fcaac63d413" containerName="revoke-keys"
Jul 24 14:12:14.093139 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.092988    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fb2761-89af-4792-a5a3-695da5903c40" containerName="main"
Jul 24 14:12:14.097514 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.097498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:14.100166 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.100134    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:12:14.100166 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.100154    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:12:14.103419 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.103396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"]
Jul 24 14:12:14.112430 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.112405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj\" (UID: \"60d8dd66-7e0c-4c40-bf68-0ff5075733a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:14.213407 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.213367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj\" (UID: \"60d8dd66-7e0c-4c40-bf68-0ff5075733a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:14.214080 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.214052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj\" (UID: \"60d8dd66-7e0c-4c40-bf68-0ff5075733a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:14.409397 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.409300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:14.549075 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:14.549053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"]
Jul 24 14:12:14.551630 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:12:14.551591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d8dd66_7e0c_4c40_bf68_0ff5075733a3.slice/crio-12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199 WatchSource:0}: Error finding container 12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199: Status 404 returned error can't find the container with id 12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199
Jul 24 14:12:15.516789 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:15.516756    2579 generic.go:356] "Generic (PLEG): container finished" podID="60d8dd66-7e0c-4c40-bf68-0ff5075733a3" containerID="35069cf081478df1cfc25a65737aedb7e56ee86447edec5fd805bc2db3b067d0" exitCode=0
Jul 24 14:12:15.517161 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:15.516797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj" event={"ID":"60d8dd66-7e0c-4c40-bf68-0ff5075733a3","Type":"ContainerDied","Data":"35069cf081478df1cfc25a65737aedb7e56ee86447edec5fd805bc2db3b067d0"}
Jul 24 14:12:15.517161 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:15.516839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj" event={"ID":"60d8dd66-7e0c-4c40-bf68-0ff5075733a3","Type":"ContainerStarted","Data":"12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199"}
Jul 24 14:12:16.647891 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:16.647865    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:16.732913 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:16.732878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca\") pod \"60d8dd66-7e0c-4c40-bf68-0ff5075733a3\" (UID: \"60d8dd66-7e0c-4c40-bf68-0ff5075733a3\") "
Jul 24 14:12:16.733225 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:16.733201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60d8dd66-7e0c-4c40-bf68-0ff5075733a3" (UID: "60d8dd66-7e0c-4c40-bf68-0ff5075733a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:12:16.833824 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:16.833744    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d8dd66-7e0c-4c40-bf68-0ff5075733a3-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:12:17.527907 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:17.527871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj" event={"ID":"60d8dd66-7e0c-4c40-bf68-0ff5075733a3","Type":"ContainerDied","Data":"12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199"}
Jul 24 14:12:17.527907 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:17.527890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"
Jul 24 14:12:17.528145 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:17.527920    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a64934394195932c1a994dba0ded4bfcb060225a95e6ef6e69227c85eb9199"
Jul 24 14:12:21.263177 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:21.263134    2579 scope.go:117] "RemoveContainer" containerID="6a9a4a395248b7be5a5f8476a756d70a95e7a14d8170c068c25394797dbebb08"
Jul 24 14:12:29.198044 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:29.198012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"]
Jul 24 14:12:29.199675 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:29.199648    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d52a8f-798nj"]
Jul 24 14:12:31.124825 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:31.124794    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d8dd66-7e0c-4c40-bf68-0ff5075733a3" path="/var/lib/kubelet/pods/60d8dd66-7e0c-4c40-bf68-0ff5075733a3/volumes"
Jul 24 14:12:48.884431 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.884393    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:12:48.884943 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.884775    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d8dd66-7e0c-4c40-bf68-0ff5075733a3" containerName="revoke-keys"
Jul 24 14:12:48.884943 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.884788    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d8dd66-7e0c-4c40-bf68-0ff5075733a3" containerName="revoke-keys"
Jul 24 14:12:48.884943 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.884862    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d8dd66-7e0c-4c40-bf68-0ff5075733a3" containerName="revoke-keys"
Jul 24 14:12:48.887990 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.887971    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:12:48.888124 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.888103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:48.890723 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.890703    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-17b50a\""
Jul 24 14:12:48.891088 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.891067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:48.891529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.891515    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-17b50a-dockercfg-vmjkr\""
Jul 24 14:12:48.901625 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.901600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:12:48.903287 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:48.903263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:12:49.013177 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.013137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74xf\" (UniqueName: \"kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.013350 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.013187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.013350 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.013207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.013350 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.013292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z75s\" (UniqueName: \"kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.114283 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.114236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z75s\" (UniqueName: \"kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.114528 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.114348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74xf\" (UniqueName: \"kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.114528 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.114382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.114528 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.114408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.115205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.115182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.115205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.115191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.123831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.123807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z75s\" (UniqueName: \"kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s\") pod \"payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.124047 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.124025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74xf\" (UniqueName: \"kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf\") pod \"payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.200639 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.200530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:49.206835 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.206796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:49.350027 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.349997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:12:49.353800 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:12:49.353757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3697839_2806_47ca_91f3_84f8ac774c8e.slice/crio-3bc0dd7deec4f850224e7c3b7663fa40f59feaa9da77de27ca9269d1835a2eda WatchSource:0}: Error finding container 3bc0dd7deec4f850224e7c3b7663fa40f59feaa9da77de27ca9269d1835a2eda: Status 404 returned error can't find the container with id 3bc0dd7deec4f850224e7c3b7663fa40f59feaa9da77de27ca9269d1835a2eda
Jul 24 14:12:49.355769 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.355752    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:12:49.373280 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.373256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:12:49.376449 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:12:49.376425    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a5c272_287f_436e_a6ea_55a5f897adbc.slice/crio-41c87e293ace1f0ac2069faf8b0724553a1bad72f2fa0460227a46e0c21e56e9 WatchSource:0}: Error finding container 41c87e293ace1f0ac2069faf8b0724553a1bad72f2fa0460227a46e0c21e56e9: Status 404 returned error can't find the container with id 41c87e293ace1f0ac2069faf8b0724553a1bad72f2fa0460227a46e0c21e56e9
Jul 24 14:12:49.680078 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.680041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" event={"ID":"b7a5c272-287f-436e-a6ea-55a5f897adbc","Type":"ContainerStarted","Data":"41c87e293ace1f0ac2069faf8b0724553a1bad72f2fa0460227a46e0c21e56e9"}
Jul 24 14:12:49.681162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:49.681138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" event={"ID":"a3697839-2806-47ca-91f3-84f8ac774c8e","Type":"ContainerStarted","Data":"3bc0dd7deec4f850224e7c3b7663fa40f59feaa9da77de27ca9269d1835a2eda"}
Jul 24 14:12:51.696583 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.696489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" event={"ID":"b7a5c272-287f-436e-a6ea-55a5f897adbc","Type":"ContainerStarted","Data":"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"}
Jul 24 14:12:51.697008 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.696590    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:12:51.698300 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.698272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" event={"ID":"a3697839-2806-47ca-91f3-84f8ac774c8e","Type":"ContainerStarted","Data":"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"}
Jul 24 14:12:51.698467 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.698362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:12:51.720715 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.720654    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" podStartSLOduration=1.828281834 podStartE2EDuration="3.720637194s" totalImagesPullingTime="1.89235536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:51.717545471 +0000 UTC m=+2311.180152344" watchObservedRunningTime="2026-07-24 14:12:51.720637194 +0000 UTC m=+2311.183244038"
Jul 24 14:12:51.738045 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:12:51.737987    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" podStartSLOduration=1.829761957 podStartE2EDuration="3.737972121s" totalImagesPullingTime="1.908210164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:12:51.736954975 +0000 UTC m=+2311.199561820" watchObservedRunningTime="2026-07-24 14:12:51.737972121 +0000 UTC m=+2311.200578966"
Jul 24 14:13:00.741142 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.741109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"]
Jul 24 14:13:00.744662 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.744642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:00.747447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.747428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:13:00.747557 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.747505    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:13:00.752221 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.752196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"]
Jul 24 14:13:00.920869 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:00.920830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-41b861-bh25w\" (UID: \"074f012f-4bf4-4c13-9d9c-2dd5ce23effd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:01.021547 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.021513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-41b861-bh25w\" (UID: \"074f012f-4bf4-4c13-9d9c-2dd5ce23effd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:01.022151 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.022132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-41b861-bh25w\" (UID: \"074f012f-4bf4-4c13-9d9c-2dd5ce23effd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:01.056242 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.056186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:01.194882 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.194801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"]
Jul 24 14:13:01.197194 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:13:01.197164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074f012f_4bf4_4c13_9d9c_2dd5ce23effd.slice/crio-cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a WatchSource:0}: Error finding container cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a: Status 404 returned error can't find the container with id cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a
Jul 24 14:13:01.750468 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.750377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w" event={"ID":"074f012f-4bf4-4c13-9d9c-2dd5ce23effd","Type":"ContainerStarted","Data":"00e2ec5545aac110531f6df737277927174c018c0962250d8d44b609514726d2"}
Jul 24 14:13:01.750468 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.750428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w" event={"ID":"074f012f-4bf4-4c13-9d9c-2dd5ce23effd","Type":"ContainerStarted","Data":"cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a"}
Jul 24 14:13:01.771859 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:01.771796    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w" podStartSLOduration=1.7717810200000002 podStartE2EDuration="1.77178102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:13:01.771198547 +0000 UTC m=+2321.233805413" watchObservedRunningTime="2026-07-24 14:13:01.77178102 +0000 UTC m=+2321.234387863"
Jul 24 14:13:02.705974 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:02.705940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:13:02.706150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:02.706022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:13:02.758175 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:02.758135    2579 generic.go:356] "Generic (PLEG): container finished" podID="074f012f-4bf4-4c13-9d9c-2dd5ce23effd" containerID="00e2ec5545aac110531f6df737277927174c018c0962250d8d44b609514726d2" exitCode=0
Jul 24 14:13:02.758638 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:02.758237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w" event={"ID":"074f012f-4bf4-4c13-9d9c-2dd5ce23effd","Type":"ContainerDied","Data":"00e2ec5545aac110531f6df737277927174c018c0962250d8d44b609514726d2"}
Jul 24 14:13:03.892709 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:03.892682    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:03.947365 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:03.947327    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca\") pod \"074f012f-4bf4-4c13-9d9c-2dd5ce23effd\" (UID: \"074f012f-4bf4-4c13-9d9c-2dd5ce23effd\") "
Jul 24 14:13:03.947740 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:03.947717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "074f012f-4bf4-4c13-9d9c-2dd5ce23effd" (UID: "074f012f-4bf4-4c13-9d9c-2dd5ce23effd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:04.048215 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:04.048137    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/074f012f-4bf4-4c13-9d9c-2dd5ce23effd-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:04.769446 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:04.769412    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"
Jul 24 14:13:04.769653 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:04.769453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w" event={"ID":"074f012f-4bf4-4c13-9d9c-2dd5ce23effd","Type":"ContainerDied","Data":"cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a"}
Jul 24 14:13:04.769653 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:04.769524    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb575130b29c9dcb3aab3dd83bbf66770057188f4cca48c4da2244dee0ccd04a"
Jul 24 14:13:20.843513 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:20.843463    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"]
Jul 24 14:13:20.846587 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:20.846552    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-41b861-bh25w"]
Jul 24 14:13:21.124624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:21.124551    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="074f012f-4bf4-4c13-9d9c-2dd5ce23effd" path="/var/lib/kubelet/pods/074f012f-4bf4-4c13-9d9c-2dd5ce23effd/volumes"
Jul 24 14:13:28.491958 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.491920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"]
Jul 24 14:13:28.492341 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.492308    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074f012f-4bf4-4c13-9d9c-2dd5ce23effd" containerName="revoke-keys"
Jul 24 14:13:28.492341 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.492319    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="074f012f-4bf4-4c13-9d9c-2dd5ce23effd" containerName="revoke-keys"
Jul 24 14:13:28.492414 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.492397    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="074f012f-4bf4-4c13-9d9c-2dd5ce23effd" containerName="revoke-keys"
Jul 24 14:13:28.502155 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.502122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:28.506204 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.506176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:13:28.506204 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.506198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:13:28.507263 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.507209    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"]
Jul 24 14:13:28.560303 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.560261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-17b50a-szzwt\" (UID: \"72f11d11-beb8-4fd7-b578-9a8059049530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:28.661024 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.660989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-17b50a-szzwt\" (UID: \"72f11d11-beb8-4fd7-b578-9a8059049530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:28.661681 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.661663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-17b50a-szzwt\" (UID: \"72f11d11-beb8-4fd7-b578-9a8059049530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:28.814746 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.814695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:28.952423 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:28.952359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"]
Jul 24 14:13:28.954889 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:13:28.954864    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f11d11_beb8_4fd7_b578_9a8059049530.slice/crio-b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae WatchSource:0}: Error finding container b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae: Status 404 returned error can't find the container with id b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae
Jul 24 14:13:29.887673 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:29.887637    2579 generic.go:356] "Generic (PLEG): container finished" podID="72f11d11-beb8-4fd7-b578-9a8059049530" containerID="ba7bdaebeb12afcca931f62c4fce3665cfb54d38a1aa3df9d1c191624d1c46a2" exitCode=0
Jul 24 14:13:29.888097 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:29.887727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt" event={"ID":"72f11d11-beb8-4fd7-b578-9a8059049530","Type":"ContainerDied","Data":"ba7bdaebeb12afcca931f62c4fce3665cfb54d38a1aa3df9d1c191624d1c46a2"}
Jul 24 14:13:29.888097 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:29.887785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt" event={"ID":"72f11d11-beb8-4fd7-b578-9a8059049530","Type":"ContainerStarted","Data":"b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae"}
Jul 24 14:13:31.024299 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.024274    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:31.084548 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.084508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca\") pod \"72f11d11-beb8-4fd7-b578-9a8059049530\" (UID: \"72f11d11-beb8-4fd7-b578-9a8059049530\") "
Jul 24 14:13:31.084864 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.084841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72f11d11-beb8-4fd7-b578-9a8059049530" (UID: "72f11d11-beb8-4fd7-b578-9a8059049530"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:31.186020 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.185932    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f11d11-beb8-4fd7-b578-9a8059049530-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:31.897886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.897859    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"
Jul 24 14:13:31.898136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.897902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt" event={"ID":"72f11d11-beb8-4fd7-b578-9a8059049530","Type":"ContainerDied","Data":"b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae"}
Jul 24 14:13:31.898136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:31.897945    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19288daf47db20416a8894f9ecabd97a5c3aaca26d3a70e73317fe7739d8cae"
Jul 24 14:13:44.336812 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.336779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:13:44.337175 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.336989    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" podUID="b7a5c272-287f-436e-a6ea-55a5f897adbc" containerName="payload-processing" containerID="cri-o://04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96" gracePeriod=30
Jul 24 14:13:44.360046 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.360016    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:13:44.360255 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.360217    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" podUID="a3697839-2806-47ca-91f3-84f8ac774c8e" containerName="payload-pre-processing" containerID="cri-o://e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2" gracePeriod=30
Jul 24 14:13:44.636207 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.636184    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:13:44.638740 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.638718    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:13:44.810419 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z75s\" (UniqueName: \"kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s\") pod \"a3697839-2806-47ca-91f3-84f8ac774c8e\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") "
Jul 24 14:13:44.810651 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810447    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume\") pod \"b7a5c272-287f-436e-a6ea-55a5f897adbc\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") "
Jul 24 14:13:44.810651 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f74xf\" (UniqueName: \"kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf\") pod \"b7a5c272-287f-436e-a6ea-55a5f897adbc\" (UID: \"b7a5c272-287f-436e-a6ea-55a5f897adbc\") "
Jul 24 14:13:44.810651 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810639    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume\") pod \"a3697839-2806-47ca-91f3-84f8ac774c8e\" (UID: \"a3697839-2806-47ca-91f3-84f8ac774c8e\") "
Jul 24 14:13:44.810972 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7a5c272-287f-436e-a6ea-55a5f897adbc" (UID: "b7a5c272-287f-436e-a6ea-55a5f897adbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:44.810972 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.810953    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3697839-2806-47ca-91f3-84f8ac774c8e" (UID: "a3697839-2806-47ca-91f3-84f8ac774c8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:13:44.813377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.813352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf" (OuterVolumeSpecName: "kube-api-access-f74xf") pod "b7a5c272-287f-436e-a6ea-55a5f897adbc" (UID: "b7a5c272-287f-436e-a6ea-55a5f897adbc"). InnerVolumeSpecName "kube-api-access-f74xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:44.813377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.813360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s" (OuterVolumeSpecName: "kube-api-access-7z75s") pod "a3697839-2806-47ca-91f3-84f8ac774c8e" (UID: "a3697839-2806-47ca-91f3-84f8ac774c8e"). InnerVolumeSpecName "kube-api-access-7z75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:13:44.911556 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.911448    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3697839-2806-47ca-91f3-84f8ac774c8e-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.911556 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.911510    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z75s\" (UniqueName: \"kubernetes.io/projected/a3697839-2806-47ca-91f3-84f8ac774c8e-kube-api-access-7z75s\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.911556 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.911522    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7a5c272-287f-436e-a6ea-55a5f897adbc-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.911556 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.911531    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74xf\" (UniqueName: \"kubernetes.io/projected/b7a5c272-287f-436e-a6ea-55a5f897adbc-kube-api-access-f74xf\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:13:44.960848 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.960817    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7a5c272-287f-436e-a6ea-55a5f897adbc" containerID="04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96" exitCode=0
Jul 24 14:13:44.961051 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.960863    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"
Jul 24 14:13:44.961051 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.960914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" event={"ID":"b7a5c272-287f-436e-a6ea-55a5f897adbc","Type":"ContainerDied","Data":"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"}
Jul 24 14:13:44.961051 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.960977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx" event={"ID":"b7a5c272-287f-436e-a6ea-55a5f897adbc","Type":"ContainerDied","Data":"41c87e293ace1f0ac2069faf8b0724553a1bad72f2fa0460227a46e0c21e56e9"}
Jul 24 14:13:44.961051 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.961007    2579 scope.go:117] "RemoveContainer" containerID="04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"
Jul 24 14:13:44.962294 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.962273    2579 generic.go:356] "Generic (PLEG): container finished" podID="a3697839-2806-47ca-91f3-84f8ac774c8e" containerID="e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2" exitCode=0
Jul 24 14:13:44.962388 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.962310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" event={"ID":"a3697839-2806-47ca-91f3-84f8ac774c8e","Type":"ContainerDied","Data":"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"}
Jul 24 14:13:44.962388 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.962333    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"
Jul 24 14:13:44.962388 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.962360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn" event={"ID":"a3697839-2806-47ca-91f3-84f8ac774c8e","Type":"ContainerDied","Data":"3bc0dd7deec4f850224e7c3b7663fa40f59feaa9da77de27ca9269d1835a2eda"}
Jul 24 14:13:44.971547 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.971522    2579 scope.go:117] "RemoveContainer" containerID="04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"
Jul 24 14:13:44.971919 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:13:44.971887    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96\": container with ID starting with 04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96 not found: ID does not exist" containerID="04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"
Jul 24 14:13:44.972004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.971931    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96"} err="failed to get container status \"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96\": rpc error: code = NotFound desc = could not find container \"04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96\": container with ID starting with 04ea2b71f89686cefbe56522e30f64e192988342034ee9c6af4f5713afbd8b96 not found: ID does not exist"
Jul 24 14:13:44.972004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.971957    2579 scope.go:117] "RemoveContainer" containerID="e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"
Jul 24 14:13:44.980205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.980188    2579 scope.go:117] "RemoveContainer" containerID="e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"
Jul 24 14:13:44.980534 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:13:44.980472    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2\": container with ID starting with e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2 not found: ID does not exist" containerID="e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"
Jul 24 14:13:44.980609 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.980524    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2"} err="failed to get container status \"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2\": rpc error: code = NotFound desc = could not find container \"e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2\": container with ID starting with e912f59f0794b3d86f26c09ee674138aace7bced5f65532fa54266a410af59b2 not found: ID does not exist"
Jul 24 14:13:44.985964 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.985940    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:13:44.991302 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:44.991280    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-17b50a-79484f74f4-rfbpx"]
Jul 24 14:13:45.002915 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:45.002889    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:13:45.006949 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:45.006925    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-17b50a-7dbd4f6bcb-rxcvn"]
Jul 24 14:13:45.125177 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:45.125143    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3697839-2806-47ca-91f3-84f8ac774c8e" path="/var/lib/kubelet/pods/a3697839-2806-47ca-91f3-84f8ac774c8e/volumes"
Jul 24 14:13:45.125518 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:45.125505    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a5c272-287f-436e-a6ea-55a5f897adbc" path="/var/lib/kubelet/pods/b7a5c272-287f-436e-a6ea-55a5f897adbc/volumes"
Jul 24 14:13:48.591393 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:48.591328    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"]
Jul 24 14:13:48.593632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:48.593605    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-17b50a-szzwt"]
Jul 24 14:13:49.124791 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:13:49.124757    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f11d11-beb8-4fd7-b578-9a8059049530" path="/var/lib/kubelet/pods/72f11d11-beb8-4fd7-b578-9a8059049530/volumes"
Jul 24 14:14:00.469682 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.469637    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:14:00.470157 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470127    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a5c272-287f-436e-a6ea-55a5f897adbc" containerName="payload-processing"
Jul 24 14:14:00.470157 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470146    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a5c272-287f-436e-a6ea-55a5f897adbc" containerName="payload-processing"
Jul 24 14:14:00.470279 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470162    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3697839-2806-47ca-91f3-84f8ac774c8e" containerName="payload-pre-processing"
Jul 24 14:14:00.470279 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470171    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3697839-2806-47ca-91f3-84f8ac774c8e" containerName="payload-pre-processing"
Jul 24 14:14:00.470279 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470192    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f11d11-beb8-4fd7-b578-9a8059049530" containerName="revoke-keys"
Jul 24 14:14:00.470279 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f11d11-beb8-4fd7-b578-9a8059049530" containerName="revoke-keys"
Jul 24 14:14:00.470503 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470307    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a5c272-287f-436e-a6ea-55a5f897adbc" containerName="payload-processing"
Jul 24 14:14:00.470503 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f11d11-beb8-4fd7-b578-9a8059049530" containerName="revoke-keys"
Jul 24 14:14:00.470503 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.470329    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3697839-2806-47ca-91f3-84f8ac774c8e" containerName="payload-pre-processing"
Jul 24 14:14:00.473377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.473355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.476197 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.476165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4cd364\""
Jul 24 14:14:00.476324 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.476300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-4cd364-dockercfg-2s9gc\""
Jul 24 14:14:00.484697 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.484669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:14:00.553294 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.553248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.553538 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.553316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqbw\" (UniqueName: \"kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.580879 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.580847    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:14:00.584445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.584420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.595107 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.595077    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:14:00.654512 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.654459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.654660 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.654514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2flb\" (UniqueName: \"kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.654660 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.654614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.654779 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.654673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqbw\" (UniqueName: \"kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.655380 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.655357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.666399 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.666369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqbw\" (UniqueName: \"kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw\") pod \"payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.755804 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.755763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2flb\" (UniqueName: \"kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.756004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.755893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.756608 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.756586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.764871 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.764846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2flb\" (UniqueName: \"kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb\") pod \"payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.785043 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.785007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:00.895815 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.895778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:00.930512 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:00.930460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:14:00.933592 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:14:00.933553    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d067f51_7d11_458e_865c_1f24163acf5e.slice/crio-5e9f983fcbc71530cbbdd73c4ba6519c5b15fec07afc6bab45bd4e28403d5e3a WatchSource:0}: Error finding container 5e9f983fcbc71530cbbdd73c4ba6519c5b15fec07afc6bab45bd4e28403d5e3a: Status 404 returned error can't find the container with id 5e9f983fcbc71530cbbdd73c4ba6519c5b15fec07afc6bab45bd4e28403d5e3a
Jul 24 14:14:01.044460 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:01.044420    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" event={"ID":"5d067f51-7d11-458e-865c-1f24163acf5e","Type":"ContainerStarted","Data":"c8ccd7894d0fb081ac7206a30feb7b7fc2c21ce6cf66b298a996a773618c26b6"}
Jul 24 14:14:01.044684 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:01.044525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" event={"ID":"5d067f51-7d11-458e-865c-1f24163acf5e","Type":"ContainerStarted","Data":"5e9f983fcbc71530cbbdd73c4ba6519c5b15fec07afc6bab45bd4e28403d5e3a"}
Jul 24 14:14:01.044684 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:01.044566    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:01.091736 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:01.091707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:14:01.093110 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:14:01.093070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02023676_3984_45d1_bc98_afa609b9e529.slice/crio-66e345ecc460277fa7317fad690c76a122b9d3e171f9cc1953bc7f138529112a WatchSource:0}: Error finding container 66e345ecc460277fa7317fad690c76a122b9d3e171f9cc1953bc7f138529112a: Status 404 returned error can't find the container with id 66e345ecc460277fa7317fad690c76a122b9d3e171f9cc1953bc7f138529112a
Jul 24 14:14:01.098889 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:01.098846    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" podStartSLOduration=1.098827398 podStartE2EDuration="1.098827398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:01.09615801 +0000 UTC m=+2380.558764878" watchObservedRunningTime="2026-07-24 14:14:01.098827398 +0000 UTC m=+2380.561434241"
Jul 24 14:14:02.050552 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:02.050512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" event={"ID":"02023676-3984-45d1-bc98-afa609b9e529","Type":"ContainerStarted","Data":"2c5b66732f662be8b071c1e24898b9a6a000f8f202faf3401a7616ed0a0acb60"}
Jul 24 14:14:02.050999 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:02.050568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" event={"ID":"02023676-3984-45d1-bc98-afa609b9e529","Type":"ContainerStarted","Data":"66e345ecc460277fa7317fad690c76a122b9d3e171f9cc1953bc7f138529112a"}
Jul 24 14:14:02.050999 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:02.050752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:02.070662 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:02.070612    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" podStartSLOduration=2.070597036 podStartE2EDuration="2.070597036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:02.069142447 +0000 UTC m=+2381.531749309" watchObservedRunningTime="2026-07-24 14:14:02.070597036 +0000 UTC m=+2381.533203886"
Jul 24 14:14:09.176561 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.176522    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:14:09.180091 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.180075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.184852 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.184832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-525009-dockercfg-k58wz\""
Jul 24 14:14:09.185173 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.185156    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-525009\""
Jul 24 14:14:09.193063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.193039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:14:09.237427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.237389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.237700 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.237660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt879\" (UniqueName: \"kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.338540 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.338465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.338733 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.338632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt879\" (UniqueName: \"kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.339203 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.339181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.349237 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.349216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt879\" (UniqueName: \"kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879\") pod \"payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.490970 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.490878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:09.627929 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:09.627903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:14:09.630778 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:14:09.630752    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8816c977_47be_4d3e_b536_e01e23fb26fb.slice/crio-6bd9ae777421f746b73d811238302e17b0122e2a82e18b982a3b1583ad4f35dd WatchSource:0}: Error finding container 6bd9ae777421f746b73d811238302e17b0122e2a82e18b982a3b1583ad4f35dd: Status 404 returned error can't find the container with id 6bd9ae777421f746b73d811238302e17b0122e2a82e18b982a3b1583ad4f35dd
Jul 24 14:14:10.090236 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:10.090200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" event={"ID":"8816c977-47be-4d3e-b536-e01e23fb26fb","Type":"ContainerStarted","Data":"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"}
Jul 24 14:14:10.090436 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:10.090261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" event={"ID":"8816c977-47be-4d3e-b536-e01e23fb26fb","Type":"ContainerStarted","Data":"6bd9ae777421f746b73d811238302e17b0122e2a82e18b982a3b1583ad4f35dd"}
Jul 24 14:14:10.090436 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:10.090298    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:12.052604 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:12.052561    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.51:9004: connect: connection refused"
Jul 24 14:14:13.058155 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:13.058127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:14:13.079428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:13.079377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" podStartSLOduration=4.079359154 podStartE2EDuration="4.079359154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:10.11409631 +0000 UTC m=+2389.576703177" watchObservedRunningTime="2026-07-24 14:14:13.079359154 +0000 UTC m=+2392.541966066"
Jul 24 14:14:21.098374 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:21.098340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:14:21.138697 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:21.138599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:14:21.152296 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:21.146762    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:14:21.299362 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:21.299330    2579 scope.go:117] "RemoveContainer" containerID="4dc72c7cf923cf48eb591ccf890b7fe133795a8739e022b3fa50bfd2a6311055"
Jul 24 14:14:22.051359 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:22.051323    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:14:23.357062 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.357031    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:14:23.361958 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.361940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.365061 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.365040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"default-dockercfg-hlqnt\""
Jul 24 14:14:23.365939 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.365923    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"kube-root-ca.crt\""
Jul 24 14:14:23.366000 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.365944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"openshift-service-ca.crt\""
Jul 24 14:14:23.366199 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.366185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"auth-test-model-4cd364-kserve-self-signed-certs\""
Jul 24 14:14:23.394155 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.394125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:14:23.463988 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.463953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.463988 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.463990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx46\" (UniqueName: \"kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.464209 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.464019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.464209 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.464081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.464209 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.464122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.464209 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.464153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565505 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565680 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565680 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565756 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565797 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx46\" (UniqueName: \"kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565843 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.565951 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.565930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.566037 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.566020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.566200 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.566144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.567978 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.567952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.568075 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.568056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.579855 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.579836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx46\" (UniqueName: \"kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46\") pod \"auth-test-model-4cd364-kserve-56d67c6795-dnmsq\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") " pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.671719 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.671600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:23.815383 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:23.815357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:14:23.817100 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:14:23.817067    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1e11063_e25e_4dd2_afb2_c6351ecf20bf.slice/crio-6117ee01eee5fac246c044febcb25c8f9cecbc1fc670f02609c7cfd599e57fbc WatchSource:0}: Error finding container 6117ee01eee5fac246c044febcb25c8f9cecbc1fc670f02609c7cfd599e57fbc: Status 404 returned error can't find the container with id 6117ee01eee5fac246c044febcb25c8f9cecbc1fc670f02609c7cfd599e57fbc
Jul 24 14:14:24.158140 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:24.158099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" event={"ID":"c1e11063-e25e-4dd2-afb2-c6351ecf20bf","Type":"ContainerStarted","Data":"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"}
Jul 24 14:14:24.158140 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:24.158145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" event={"ID":"c1e11063-e25e-4dd2-afb2-c6351ecf20bf","Type":"ContainerStarted","Data":"6117ee01eee5fac246c044febcb25c8f9cecbc1fc670f02609c7cfd599e57fbc"}
Jul 24 14:14:33.671914 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:33.671872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:33.671914 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:33.671919    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:33.689620 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:33.689593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:33.715466 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:33.715408    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" podStartSLOduration=10.715390008 podStartE2EDuration="10.715390008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:24.199215673 +0000 UTC m=+2403.661822519" watchObservedRunningTime="2026-07-24 14:14:33.715390008 +0000 UTC m=+2413.177996864"
Jul 24 14:14:34.217873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:34.217843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:14:52.245967 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.245882    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:14:52.250493 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.250456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.253797 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.253773    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"kube-root-ca.crt\""
Jul 24 14:14:52.254896 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.254874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"default-dockercfg-qfbwm\""
Jul 24 14:14:52.255187 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.255163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"openshift-service-ca.crt\""
Jul 24 14:14:52.255244 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.255198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"auth-test-model-525009-kserve-self-signed-certs\""
Jul 24 14:14:52.279247 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.279219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:14:52.325923 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.325886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.325923 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.325923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.326134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.325960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxtzx\" (UniqueName: \"kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.326134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.326048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.326134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.326067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.326134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.326120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427512 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427512 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427762 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427762 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427762 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427762 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxtzx\" (UniqueName: \"kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427964 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.427964 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.428067 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.427979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.430069 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.430044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.430439 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.430419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.439874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.439848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxtzx\" (UniqueName: \"kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx\") pod \"auth-test-model-525009-kserve-55dd9cdf5f-cnhmb\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") " pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.560269 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.560230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:14:52.711887 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:52.711797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:14:52.714600 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:14:52.714568    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0019d43c_45c8_4e73_9692_f4f438032c6b.slice/crio-da94e29c44399c5fdf694c391f6cf512f97e8f74a4d14b7653f49fb711addaf0 WatchSource:0}: Error finding container da94e29c44399c5fdf694c391f6cf512f97e8f74a4d14b7653f49fb711addaf0: Status 404 returned error can't find the container with id da94e29c44399c5fdf694c391f6cf512f97e8f74a4d14b7653f49fb711addaf0
Jul 24 14:14:53.299795 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:53.299756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" event={"ID":"0019d43c-45c8-4e73-9692-f4f438032c6b","Type":"ContainerStarted","Data":"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"}
Jul 24 14:14:53.299795 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:53.299801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" event={"ID":"0019d43c-45c8-4e73-9692-f4f438032c6b","Type":"ContainerStarted","Data":"da94e29c44399c5fdf694c391f6cf512f97e8f74a4d14b7653f49fb711addaf0"}
Jul 24 14:14:53.321785 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:14:53.321733    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" podStartSLOduration=1.32171794 podStartE2EDuration="1.32171794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:14:53.319769228 +0000 UTC m=+2432.782376265" watchObservedRunningTime="2026-07-24 14:14:53.32171794 +0000 UTC m=+2432.784324800"
Jul 24 14:15:00.144918 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.144875    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"]
Jul 24 14:15:00.148505 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.148465    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"]
Jul 24 14:15:00.148609 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.148589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"
Jul 24 14:15:00.151911 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.151891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"
Jul 24 14:15:00.155336 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.155313    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:15:00.159564 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.159450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"]
Jul 24 14:15:00.162332 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.162300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"]
Jul 24 14:15:00.163500 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.163456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"
Jul 24 14:15:00.172159 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.172137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"
Jul 24 14:15:00.316958 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.316927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"]
Jul 24 14:15:00.319900 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:15:00.319870    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14685f0_8d12_446c_9952_4cc8cfba7184.slice/crio-dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed WatchSource:0}: Error finding container dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed: Status 404 returned error can't find the container with id dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed
Jul 24 14:15:00.335706 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.335674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692" event={"ID":"b14685f0-8d12-446c-9952-4cc8cfba7184","Type":"ContainerStarted","Data":"dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed"}
Jul 24 14:15:00.342257 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:00.342235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"]
Jul 24 14:15:00.345156 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:15:00.345129    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8acb7a_d03a_4ce8_8883_b33f3e5e2db9.slice/crio-a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32 WatchSource:0}: Error finding container a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32: Status 404 returned error can't find the container with id a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32
Jul 24 14:15:01.342117 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:01.342087    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9" containerID="b51a5588fe7793239d479c5cf592ce98861f05884d5d505c77a784ee93b4a049" exitCode=0
Jul 24 14:15:01.342564 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:01.342169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv" event={"ID":"1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9","Type":"ContainerDied","Data":"b51a5588fe7793239d479c5cf592ce98861f05884d5d505c77a784ee93b4a049"}
Jul 24 14:15:01.342564 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:01.342223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv" event={"ID":"1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9","Type":"ContainerStarted","Data":"a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32"}
Jul 24 14:15:01.343624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:01.343602    2579 generic.go:356] "Generic (PLEG): container finished" podID="b14685f0-8d12-446c-9952-4cc8cfba7184" containerID="378d76a8def1190f25fa9100e6274eeb1360f63352d2eb8afa2fb41f2ec045fd" exitCode=0
Jul 24 14:15:01.343722 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:01.343630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692" event={"ID":"b14685f0-8d12-446c-9952-4cc8cfba7184","Type":"ContainerDied","Data":"378d76a8def1190f25fa9100e6274eeb1360f63352d2eb8afa2fb41f2ec045fd"}
Jul 24 14:15:02.487830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:02.487802    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"
Jul 24 14:15:02.512228 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:02.512199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"
Jul 24 14:15:02.560530 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:02.560501    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:15:02.560695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:02.560580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:15:02.572959 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:02.572933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:15:03.354467 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.354438    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"
Jul 24 14:15:03.354467 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.354465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692" event={"ID":"b14685f0-8d12-446c-9952-4cc8cfba7184","Type":"ContainerDied","Data":"dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed"}
Jul 24 14:15:03.354716 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.354536    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc880c5dcca7e36e24a12a55656a7bf753614a284adadebf76a773f6bd23d7ed"
Jul 24 14:15:03.355901 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.355873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv"
Jul 24 14:15:03.356021 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.355905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748375-n6wgv" event={"ID":"1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9","Type":"ContainerDied","Data":"a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32"}
Jul 24 14:15:03.356021 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.355940    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4831736b6a8a39d468cadd72193b956d8a985377b9f487a5380001fd27b1c32"
Jul 24 14:15:03.367963 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:15:03.367937    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:17:02.459504 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:02.459447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:17:03.446143 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:03.446107    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:17:17.478914 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.478847    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" podUID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" containerName="main" containerID="cri-o://b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb" gracePeriod=45
Jul 24 14:17:17.731526 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.731447    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:17:17.885062 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885062 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885153    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hx46\" (UniqueName: \"kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885205    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885443 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home\") pod \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\" (UID: \"c1e11063-e25e-4dd2-afb2-c6351ecf20bf\") "
Jul 24 14:17:17.885443 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:17.885443 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:17.885707 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885680    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home" (OuterVolumeSpecName: "home") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:17.885846 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885777    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:17.885846 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885801    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:17.885846 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.885815    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:17.887631 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.887605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm" (OuterVolumeSpecName: "dshm") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:17.887740 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.887667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46" (OuterVolumeSpecName: "kube-api-access-7hx46") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "kube-api-access-7hx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:17:17.887740 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.887668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1e11063-e25e-4dd2-afb2-c6351ecf20bf" (UID: "c1e11063-e25e-4dd2-afb2-c6351ecf20bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:17:17.986552 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.986420    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:17.986552 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.986468    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hx46\" (UniqueName: \"kubernetes.io/projected/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-kube-api-access-7hx46\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:17.986552 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:17.986508    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e11063-e25e-4dd2-afb2-c6351ecf20bf-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.020786 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.020748    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" containerID="b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb" exitCode=0
Jul 24 14:17:18.020965 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.020803    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"
Jul 24 14:17:18.020965 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.020840    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" event={"ID":"c1e11063-e25e-4dd2-afb2-c6351ecf20bf","Type":"ContainerDied","Data":"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"}
Jul 24 14:17:18.020965 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.020877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq" event={"ID":"c1e11063-e25e-4dd2-afb2-c6351ecf20bf","Type":"ContainerDied","Data":"6117ee01eee5fac246c044febcb25c8f9cecbc1fc670f02609c7cfd599e57fbc"}
Jul 24 14:17:18.020965 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.020894    2579 scope.go:117] "RemoveContainer" containerID="b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"
Jul 24 14:17:18.032513 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.032496    2579 scope.go:117] "RemoveContainer" containerID="b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"
Jul 24 14:17:18.032791 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:18.032771    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb\": container with ID starting with b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb not found: ID does not exist" containerID="b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"
Jul 24 14:17:18.032854 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.032802    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb"} err="failed to get container status \"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb\": rpc error: code = NotFound desc = could not find container \"b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb\": container with ID starting with b34002c09a1c13930bdd2355aa39b970b7742bb8f529e80ed23361f8ebe971cb not found: ID does not exist"
Jul 24 14:17:18.047170 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.047133    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:17:18.048820 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.048798    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/auth-test-model-4cd364-kserve-56d67c6795-dnmsq"]
Jul 24 14:17:18.465628 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.465587    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" podUID="0019d43c-45c8-4e73-9692-f4f438032c6b" containerName="main" containerID="cri-o://310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8" gracePeriod=45
Jul 24 14:17:18.722517 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.722436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:17:18.894006 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.893972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxtzx\" (UniqueName: \"kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894006 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894007    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894232 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894232 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894232 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894232 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894162    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs\") pod \"0019d43c-45c8-4e73-9692-f4f438032c6b\" (UID: \"0019d43c-45c8-4e73-9692-f4f438032c6b\") "
Jul 24 14:17:18.894445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home" (OuterVolumeSpecName: "home") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:18.894445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:18.894445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.894381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:18.896424 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.896396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm" (OuterVolumeSpecName: "dshm") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:17:18.896588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.896459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:17:18.896588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.896461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx" (OuterVolumeSpecName: "kube-api-access-zxtzx") pod "0019d43c-45c8-4e73-9692-f4f438032c6b" (UID: "0019d43c-45c8-4e73-9692-f4f438032c6b"). InnerVolumeSpecName "kube-api-access-zxtzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:17:18.995162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995082    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.995162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995106    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0019d43c-45c8-4e73-9692-f4f438032c6b-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.995162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995118    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxtzx\" (UniqueName: \"kubernetes.io/projected/0019d43c-45c8-4e73-9692-f4f438032c6b-kube-api-access-zxtzx\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.995162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995127    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.995162 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995164    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:18.995427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:18.995172    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0019d43c-45c8-4e73-9692-f4f438032c6b-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:17:19.027164 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.027128    2579 generic.go:356] "Generic (PLEG): container finished" podID="0019d43c-45c8-4e73-9692-f4f438032c6b" containerID="310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8" exitCode=0
Jul 24 14:17:19.027301 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.027180    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"
Jul 24 14:17:19.027301 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.027205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" event={"ID":"0019d43c-45c8-4e73-9692-f4f438032c6b","Type":"ContainerDied","Data":"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"}
Jul 24 14:17:19.027301 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.027251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb" event={"ID":"0019d43c-45c8-4e73-9692-f4f438032c6b","Type":"ContainerDied","Data":"da94e29c44399c5fdf694c391f6cf512f97e8f74a4d14b7653f49fb711addaf0"}
Jul 24 14:17:19.027301 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.027270    2579 scope.go:117] "RemoveContainer" containerID="310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"
Jul 24 14:17:19.037100 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.037081    2579 scope.go:117] "RemoveContainer" containerID="310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"
Jul 24 14:17:19.037342 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:19.037324    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8\": container with ID starting with 310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8 not found: ID does not exist" containerID="310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"
Jul 24 14:17:19.037396 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.037349    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8"} err="failed to get container status \"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8\": rpc error: code = NotFound desc = could not find container \"310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8\": container with ID starting with 310d4b0b979b80d742dd8b2250cb12c5c08e9a6fd4b48d9dbf54d3393acea3b8 not found: ID does not exist"
Jul 24 14:17:19.066430 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.066391    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:17:19.068056 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.068037    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-525009/auth-test-model-525009-kserve-55dd9cdf5f-cnhmb"]
Jul 24 14:17:19.124325 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.124290    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0019d43c-45c8-4e73-9692-f4f438032c6b" path="/var/lib/kubelet/pods/0019d43c-45c8-4e73-9692-f4f438032c6b/volumes"
Jul 24 14:17:19.124803 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:19.124782    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" path="/var/lib/kubelet/pods/c1e11063-e25e-4dd2-afb2-c6351ecf20bf/volumes"
Jul 24 14:17:36.754290 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754255    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754608    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9" containerName="cleanup"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754619    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9" containerName="cleanup"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0019d43c-45c8-4e73-9692-f4f438032c6b" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0019d43c-45c8-4e73-9692-f4f438032c6b" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14685f0-8d12-446c-9952-4cc8cfba7184" containerName="cleanup"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754649    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14685f0-8d12-446c-9952-4cc8cfba7184" containerName="cleanup"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754663    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754725    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0019d43c-45c8-4e73-9692-f4f438032c6b" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1e11063-e25e-4dd2-afb2-c6351ecf20bf" containerName="main"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754740    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14685f0-8d12-446c-9952-4cc8cfba7184" containerName="cleanup"
Jul 24 14:17:36.754771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.754745    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9" containerName="cleanup"
Jul 24 14:17:36.757567 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.757551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.760732 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:36.760705    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"sub-test-model-525009-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-525009\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"sub-test-model-525009-kserve-self-signed-certs\"" type="*v1.Secret"
Jul 24 14:17:36.760852 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:36.760739    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-qfbwm\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-525009\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"default-dockercfg-qfbwm\"" type="*v1.Secret"
Jul 24 14:17:36.761201 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:36.761176    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-525009\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 24 14:17:36.761833 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:36.761810    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-525009\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.761956 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:36.761930    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-b-525009\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 24 14:17:36.799256 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.799233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:17:36.848720 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.848886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.848886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.848886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.848886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.848886 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.848869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949631 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.949828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.949754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.950063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.950021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.950146 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.950120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.950196 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.950152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:36.952033 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:36.952013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:37.950805 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:37.950767    2579 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Jul 24 14:17:37.951214 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:37.950852    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs podName:29bddf63-f36f-4d81-b6b7-0bb099fe15fe nodeName:}" failed. No retries permitted until 2026-07-24 14:17:38.450833525 +0000 UTC m=+2597.913440349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs") pod "sub-test-model-525009-kserve-7c9fc8cf65-v89md" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe") : failed to sync secret cache: timed out waiting for the condition
Jul 24 14:17:37.957934 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:37.957905    2579 projected.go:291] Couldn't get configMap ai-tenant-e2e-shared-b-525009/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 24 14:17:38.015172 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.015142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"openshift-service-ca.crt\""
Jul 24 14:17:38.018646 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:38.018628    2579 projected.go:196] Error preparing data for projected volume kube-api-access-d2fbx for pod ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md: failed to sync configmap cache: timed out waiting for the condition
Jul 24 14:17:38.018709 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:17:38.018694    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx podName:29bddf63-f36f-4d81-b6b7-0bb099fe15fe nodeName:}" failed. No retries permitted until 2026-07-24 14:17:38.518677942 +0000 UTC m=+2597.981284765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d2fbx" (UniqueName: "kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx") pod "sub-test-model-525009-kserve-7c9fc8cf65-v89md" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe") : failed to sync configmap cache: timed out waiting for the condition
Jul 24 14:17:38.133211 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.133180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"kube-root-ca.crt\""
Jul 24 14:17:38.207537 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.207434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"default-dockercfg-qfbwm\""
Jul 24 14:17:38.257595 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.257565    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-525009\"/\"sub-test-model-525009-kserve-self-signed-certs\""
Jul 24 14:17:38.464378 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.464283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:38.466828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.466806    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:38.565305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.565260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:38.568057 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.568022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") pod \"sub-test-model-525009-kserve-7c9fc8cf65-v89md\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") " pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:38.866704 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:38.866649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:39.006575 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:39.006547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:17:39.011595 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:17:39.011564    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29bddf63_f36f_4d81_b6b7_0bb099fe15fe.slice/crio-c1a05d9ad8615ee2729264a35ad9c57d74b793f06c3a73341ecc157ec61fb71d WatchSource:0}: Error finding container c1a05d9ad8615ee2729264a35ad9c57d74b793f06c3a73341ecc157ec61fb71d: Status 404 returned error can't find the container with id c1a05d9ad8615ee2729264a35ad9c57d74b793f06c3a73341ecc157ec61fb71d
Jul 24 14:17:39.127646 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:39.127561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" event={"ID":"29bddf63-f36f-4d81-b6b7-0bb099fe15fe","Type":"ContainerStarted","Data":"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"}
Jul 24 14:17:39.127646 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:39.127601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" event={"ID":"29bddf63-f36f-4d81-b6b7-0bb099fe15fe","Type":"ContainerStarted","Data":"c1a05d9ad8615ee2729264a35ad9c57d74b793f06c3a73341ecc157ec61fb71d"}
Jul 24 14:17:39.164675 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:39.164621    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" podStartSLOduration=3.164604193 podStartE2EDuration="3.164604193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:17:39.163818953 +0000 UTC m=+2598.626425810" watchObservedRunningTime="2026-07-24 14:17:39.164604193 +0000 UTC m=+2598.627211036"
Jul 24 14:17:48.867388 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:48.867350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:48.867814 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:48.867403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:48.880043 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:48.880017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:17:49.190570 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:17:49.190467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:18:21.359068 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:21.359034    2579 scope.go:117] "RemoveContainer" containerID="35069cf081478df1cfc25a65737aedb7e56ee86447edec5fd805bc2db3b067d0"
Jul 24 14:18:21.367724 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:21.367701    2579 scope.go:117] "RemoveContainer" containerID="2f03ea6b1c3a32cb53b3ada2df79b53d0eb5416f53e6f7e8fe8bf2215f68af5b"
Jul 24 14:18:29.610145 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:29.610107    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:18:44.629519 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.629460    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" containerName="main" containerID="cri-o://20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5" gracePeriod=45
Jul 24 14:18:44.877426 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.877401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:18:44.961028 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.960947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961028 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961224 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961068    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961224 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961224 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961224 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") pod \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\" (UID: \"29bddf63-f36f-4d81-b6b7-0bb099fe15fe\") "
Jul 24 14:18:44.961382 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:18:44.961382 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home" (OuterVolumeSpecName: "home") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:18:44.961459 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:18:44.961539 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961468    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:44.961539 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.961513    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:44.963445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.963422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm" (OuterVolumeSpecName: "dshm") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:18:44.963445 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.963441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:18:44.963592 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:44.963561    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx" (OuterVolumeSpecName: "kube-api-access-d2fbx") pod "29bddf63-f36f-4d81-b6b7-0bb099fe15fe" (UID: "29bddf63-f36f-4d81-b6b7-0bb099fe15fe"). InnerVolumeSpecName "kube-api-access-d2fbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:18:45.062342 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.062299    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:45.062342 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.062330    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2fbx\" (UniqueName: \"kubernetes.io/projected/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-kube-api-access-d2fbx\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:45.062342 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.062344    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:45.062622 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.062356    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29bddf63-f36f-4d81-b6b7-0bb099fe15fe-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:18:45.460871 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.460834    2579 generic.go:356] "Generic (PLEG): container finished" podID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" containerID="20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5" exitCode=0
Jul 24 14:18:45.461073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.460884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"
Jul 24 14:18:45.461073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.460901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" event={"ID":"29bddf63-f36f-4d81-b6b7-0bb099fe15fe","Type":"ContainerDied","Data":"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"}
Jul 24 14:18:45.461073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.460941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md" event={"ID":"29bddf63-f36f-4d81-b6b7-0bb099fe15fe","Type":"ContainerDied","Data":"c1a05d9ad8615ee2729264a35ad9c57d74b793f06c3a73341ecc157ec61fb71d"}
Jul 24 14:18:45.461073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.460964    2579 scope.go:117] "RemoveContainer" containerID="20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"
Jul 24 14:18:45.469460 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.469442    2579 scope.go:117] "RemoveContainer" containerID="20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"
Jul 24 14:18:45.469719 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:18:45.469700    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5\": container with ID starting with 20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5 not found: ID does not exist" containerID="20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"
Jul 24 14:18:45.469778 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.469728    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5"} err="failed to get container status \"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5\": rpc error: code = NotFound desc = could not find container \"20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5\": container with ID starting with 20da8df262edef51a9182c7c6a789cc73704e57f0d3904eb446cff851c35cab5 not found: ID does not exist"
Jul 24 14:18:45.484212 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.484183    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:18:45.488955 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:45.488933    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-525009/sub-test-model-525009-kserve-7c9fc8cf65-v89md"]
Jul 24 14:18:47.125871 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.125837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" path="/var/lib/kubelet/pods/29bddf63-f36f-4d81-b6b7-0bb099fe15fe/volumes"
Jul 24 14:18:47.224840 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.224809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:18:47.225179 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.225167    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" containerName="main"
Jul 24 14:18:47.225240 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.225181    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" containerName="main"
Jul 24 14:18:47.225278 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.225248    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bddf63-f36f-4d81-b6b7-0bb099fe15fe" containerName="main"
Jul 24 14:18:47.229738 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.229714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.233121 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.233084    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"kube-root-ca.crt\""
Jul 24 14:18:47.233237 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.233161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"default-dockercfg-hlqnt\""
Jul 24 14:18:47.234257 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.234237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"rate-test-model-4cd364-kserve-self-signed-certs\""
Jul 24 14:18:47.234821 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.234805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cd364\"/\"openshift-service-ca.crt\""
Jul 24 14:18:47.254572 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.254545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:18:47.385732 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.385732 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.385732 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.385948 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjld\" (UniqueName: \"kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.385948 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.385948 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.385912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.486801 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjld\" (UniqueName: \"kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487004 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.486968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487261 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.487244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487333 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.487308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.487385 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.487353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.489401 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.489380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.489679 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.489662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.495846 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.495824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjld\" (UniqueName: \"kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld\") pod \"rate-test-model-4cd364-kserve-65bbb889bc-djnmz\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") " pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.538905 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.538864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:47.684172 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:47.684147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:18:47.686162 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:18:47.686130    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a6d83ae_d192_4fc5_af28_0ad5666c0945.slice/crio-24096de0a64bea81da3df113876f737ff18e303e4832ab86740066604eab192c WatchSource:0}: Error finding container 24096de0a64bea81da3df113876f737ff18e303e4832ab86740066604eab192c: Status 404 returned error can't find the container with id 24096de0a64bea81da3df113876f737ff18e303e4832ab86740066604eab192c
Jul 24 14:18:48.477182 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:48.477148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" event={"ID":"2a6d83ae-d192-4fc5-af28-0ad5666c0945","Type":"ContainerStarted","Data":"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"}
Jul 24 14:18:48.477182 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:48.477187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" event={"ID":"2a6d83ae-d192-4fc5-af28-0ad5666c0945","Type":"ContainerStarted","Data":"24096de0a64bea81da3df113876f737ff18e303e4832ab86740066604eab192c"}
Jul 24 14:18:48.497756 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:48.497698    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" podStartSLOduration=1.497681813 podStartE2EDuration="1.497681813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:48.496703121 +0000 UTC m=+2667.959309965" watchObservedRunningTime="2026-07-24 14:18:48.497681813 +0000 UTC m=+2667.960288657"
Jul 24 14:18:57.539923 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:57.539885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:57.539923 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:57.539930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:57.552607 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:57.552582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:18:58.532790 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:18:58.532760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:19:21.170858 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:21.170757    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:19:21.179308 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:21.179286    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:19:21.389725 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:21.389692    2579 scope.go:117] "RemoveContainer" containerID="00e2ec5545aac110531f6df737277927174c018c0962250d8d44b609514726d2"
Jul 24 14:19:57.791365 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:57.791325    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:19:59.788523 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.788468    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319602203184"]
Jul 24 14:19:59.792023 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.791998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:19:59.794917 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.794897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdjzj\""
Jul 24 14:19:59.799803 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.799776    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319602203184"]
Jul 24 14:19:59.826757 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.826719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxhd\" (UniqueName: \"kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd\") pod \"test-curl-7612-140319602203184\" (UID: \"542082cb-1503-4a98-9b08-a1956a09ded5\") " pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:19:59.927354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.927312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxhd\" (UniqueName: \"kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd\") pod \"test-curl-7612-140319602203184\" (UID: \"542082cb-1503-4a98-9b08-a1956a09ded5\") " pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:19:59.936978 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:19:59.936956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxhd\" (UniqueName: \"kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd\") pod \"test-curl-7612-140319602203184\" (UID: \"542082cb-1503-4a98-9b08-a1956a09ded5\") " pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:20:00.102202 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:00.102113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:20:00.236052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:00.236029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319602203184"]
Jul 24 14:20:00.238692 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:00.238662    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod542082cb_1503_4a98_9b08_a1956a09ded5.slice/crio-47562f5a1c1559e6ea6431113c0e92890177e0fad293e66306ccc5ffc8f1e450 WatchSource:0}: Error finding container 47562f5a1c1559e6ea6431113c0e92890177e0fad293e66306ccc5ffc8f1e450: Status 404 returned error can't find the container with id 47562f5a1c1559e6ea6431113c0e92890177e0fad293e66306ccc5ffc8f1e450
Jul 24 14:20:00.239978 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:00.239950    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:20:00.812428 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:00.812379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319602203184" event={"ID":"542082cb-1503-4a98-9b08-a1956a09ded5","Type":"ContainerStarted","Data":"47562f5a1c1559e6ea6431113c0e92890177e0fad293e66306ccc5ffc8f1e450"}
Jul 24 14:20:01.818333 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:01.818302    2579 generic.go:356] "Generic (PLEG): container finished" podID="542082cb-1503-4a98-9b08-a1956a09ded5" containerID="d82a83b6365dc9792ae82d1f9700a91aec2f250183cc82676a30146ae2399c9e" exitCode=0
Jul 24 14:20:01.818748 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:01.818389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319602203184" event={"ID":"542082cb-1503-4a98-9b08-a1956a09ded5","Type":"ContainerDied","Data":"d82a83b6365dc9792ae82d1f9700a91aec2f250183cc82676a30146ae2399c9e"}
Jul 24 14:20:02.954492 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:02.954457    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:20:03.008960 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.008928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319602203184_542082cb-1503-4a98-9b08-a1956a09ded5/test-curl-7612-140319602203184/0.log"
Jul 24 14:20:03.032890 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.032856    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319602203184"]
Jul 24 14:20:03.039905 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.039882    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319602203184"]
Jul 24 14:20:03.056789 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.056768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljxhd\" (UniqueName: \"kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd\") pod \"542082cb-1503-4a98-9b08-a1956a09ded5\" (UID: \"542082cb-1503-4a98-9b08-a1956a09ded5\") "
Jul 24 14:20:03.059067 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.059038    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd" (OuterVolumeSpecName: "kube-api-access-ljxhd") pod "542082cb-1503-4a98-9b08-a1956a09ded5" (UID: "542082cb-1503-4a98-9b08-a1956a09ded5"). InnerVolumeSpecName "kube-api-access-ljxhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:03.124817 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.124737    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542082cb-1503-4a98-9b08-a1956a09ded5" path="/var/lib/kubelet/pods/542082cb-1503-4a98-9b08-a1956a09ded5/volumes"
Jul 24 14:20:03.158429 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.158401    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljxhd\" (UniqueName: \"kubernetes.io/projected/542082cb-1503-4a98-9b08-a1956a09ded5-kube-api-access-ljxhd\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:03.175776 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.175738    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319602315696"]
Jul 24 14:20:03.176103 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.176091    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542082cb-1503-4a98-9b08-a1956a09ded5" containerName="test-curl-7612-140319602203184"
Jul 24 14:20:03.176150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.176105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="542082cb-1503-4a98-9b08-a1956a09ded5" containerName="test-curl-7612-140319602203184"
Jul 24 14:20:03.176249 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.176162    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="542082cb-1503-4a98-9b08-a1956a09ded5" containerName="test-curl-7612-140319602203184"
Jul 24 14:20:03.179062 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.179046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:03.188623 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.188602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319602315696"]
Jul 24 14:20:03.259627 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.259592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgjc\" (UniqueName: \"kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc\") pod \"test-curl-7612-140319602315696\" (UID: \"8bd509e6-cb8b-4fd8-9344-52ed15707f9e\") " pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:03.360887 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.360852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgjc\" (UniqueName: \"kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc\") pod \"test-curl-7612-140319602315696\" (UID: \"8bd509e6-cb8b-4fd8-9344-52ed15707f9e\") " pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:03.371070 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.371040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgjc\" (UniqueName: \"kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc\") pod \"test-curl-7612-140319602315696\" (UID: \"8bd509e6-cb8b-4fd8-9344-52ed15707f9e\") " pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:03.488505 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.488387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:03.622895 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.622871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319602315696"]
Jul 24 14:20:03.624920 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:03.624889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bd509e6_cb8b_4fd8_9344_52ed15707f9e.slice/crio-0f24f37a7d989d36474e63111021d6b0d365705c679a1d208e8d16a2c09de3d2 WatchSource:0}: Error finding container 0f24f37a7d989d36474e63111021d6b0d365705c679a1d208e8d16a2c09de3d2: Status 404 returned error can't find the container with id 0f24f37a7d989d36474e63111021d6b0d365705c679a1d208e8d16a2c09de3d2
Jul 24 14:20:03.829245 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.829213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602203184"
Jul 24 14:20:03.829245 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.829236    2579 scope.go:117] "RemoveContainer" containerID="d82a83b6365dc9792ae82d1f9700a91aec2f250183cc82676a30146ae2399c9e"
Jul 24 14:20:03.830599 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:03.830511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319602315696" event={"ID":"8bd509e6-cb8b-4fd8-9344-52ed15707f9e","Type":"ContainerStarted","Data":"0f24f37a7d989d36474e63111021d6b0d365705c679a1d208e8d16a2c09de3d2"}
Jul 24 14:20:04.836906 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:04.836874    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" containerID="7ac402939ad74dfbee986d5f4287ef74506fcfe8abcf99b20637a3cf96467064" exitCode=0
Jul 24 14:20:04.837359 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:04.836957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319602315696" event={"ID":"8bd509e6-cb8b-4fd8-9344-52ed15707f9e","Type":"ContainerDied","Data":"7ac402939ad74dfbee986d5f4287ef74506fcfe8abcf99b20637a3cf96467064"}
Jul 24 14:20:05.978686 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:05.978659    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:06.003222 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.003198    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319602315696_8bd509e6-cb8b-4fd8-9344-52ed15707f9e/test-curl-7612-140319602315696/0.log"
Jul 24 14:20:06.028112 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.028081    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319602315696"]
Jul 24 14:20:06.035573 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.035542    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319602315696"]
Jul 24 14:20:06.083142 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.083098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kgjc\" (UniqueName: \"kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc\") pod \"8bd509e6-cb8b-4fd8-9344-52ed15707f9e\" (UID: \"8bd509e6-cb8b-4fd8-9344-52ed15707f9e\") "
Jul 24 14:20:06.085317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.085294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc" (OuterVolumeSpecName: "kube-api-access-9kgjc") pod "8bd509e6-cb8b-4fd8-9344-52ed15707f9e" (UID: "8bd509e6-cb8b-4fd8-9344-52ed15707f9e"). InnerVolumeSpecName "kube-api-access-9kgjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:06.184506 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.184399    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgjc\" (UniqueName: \"kubernetes.io/projected/8bd509e6-cb8b-4fd8-9344-52ed15707f9e-kube-api-access-9kgjc\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:06.209134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.209102    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:06.209499 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.209469    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" containerName="test-curl-7612-140319602315696"
Jul 24 14:20:06.209499 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.209499    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" containerName="test-curl-7612-140319602315696"
Jul 24 14:20:06.209580 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.209567    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" containerName="test-curl-7612-140319602315696"
Jul 24 14:20:06.212504 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.212469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:06.219850 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.219816    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:06.285313 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.285273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzq4\" (UniqueName: \"kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4\") pod \"test-curl-7612-140319604496400\" (UID: \"615bfc17-1ff6-41b8-9be1-7520d443a622\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:06.386689 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.386651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzq4\" (UniqueName: \"kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4\") pod \"test-curl-7612-140319604496400\" (UID: \"615bfc17-1ff6-41b8-9be1-7520d443a622\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:06.397066 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.397037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzq4\" (UniqueName: \"kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4\") pod \"test-curl-7612-140319604496400\" (UID: \"615bfc17-1ff6-41b8-9be1-7520d443a622\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:06.522209 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.522172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:06.679697 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.679669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:06.681618 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:06.681592    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615bfc17_1ff6_41b8_9be1_7520d443a622.slice/crio-0c2b8b4dea657e985b290e6d5e065b3280a8f08bdc6b34cb19413e987a88f657 WatchSource:0}: Error finding container 0c2b8b4dea657e985b290e6d5e065b3280a8f08bdc6b34cb19413e987a88f657: Status 404 returned error can't find the container with id 0c2b8b4dea657e985b290e6d5e065b3280a8f08bdc6b34cb19413e987a88f657
Jul 24 14:20:06.848097 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.847995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604496400" event={"ID":"615bfc17-1ff6-41b8-9be1-7520d443a622","Type":"ContainerStarted","Data":"0c2b8b4dea657e985b290e6d5e065b3280a8f08bdc6b34cb19413e987a88f657"}
Jul 24 14:20:06.849115 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.849097    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f24f37a7d989d36474e63111021d6b0d365705c679a1d208e8d16a2c09de3d2"
Jul 24 14:20:06.849231 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:06.849125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:06.859941 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:20:06.859913    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-140319602315696\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" pod="opendatahub/test-curl-7612-140319602315696"
Jul 24 14:20:07.130577 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:07.130472    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd509e6-cb8b-4fd8-9344-52ed15707f9e" path="/var/lib/kubelet/pods/8bd509e6-cb8b-4fd8-9344-52ed15707f9e/volumes"
Jul 24 14:20:07.854652 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:07.854612    2579 generic.go:356] "Generic (PLEG): container finished" podID="615bfc17-1ff6-41b8-9be1-7520d443a622" containerID="b2e1f1c05f8d3f4b9bc225a6ec020193622f80776068423898de25ceffc4c633" exitCode=0
Jul 24 14:20:07.854838 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:07.854707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604496400" event={"ID":"615bfc17-1ff6-41b8-9be1-7520d443a622","Type":"ContainerDied","Data":"b2e1f1c05f8d3f4b9bc225a6ec020193622f80776068423898de25ceffc4c633"}
Jul 24 14:20:08.992797 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:08.992773    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.024083 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.024056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319604496400_615bfc17-1ff6-41b8-9be1-7520d443a622/test-curl-7612-140319604496400/0.log"
Jul 24 14:20:09.052050 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.052018    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:09.055666 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.055641    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:09.112001 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.111922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctzq4\" (UniqueName: \"kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4\") pod \"615bfc17-1ff6-41b8-9be1-7520d443a622\" (UID: \"615bfc17-1ff6-41b8-9be1-7520d443a622\") "
Jul 24 14:20:09.114164 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.114134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4" (OuterVolumeSpecName: "kube-api-access-ctzq4") pod "615bfc17-1ff6-41b8-9be1-7520d443a622" (UID: "615bfc17-1ff6-41b8-9be1-7520d443a622"). InnerVolumeSpecName "kube-api-access-ctzq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:09.124089 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.124065    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615bfc17-1ff6-41b8-9be1-7520d443a622" path="/var/lib/kubelet/pods/615bfc17-1ff6-41b8-9be1-7520d443a622/volumes"
Jul 24 14:20:09.181789 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.181765    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:09.182124 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.182111    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615bfc17-1ff6-41b8-9be1-7520d443a622" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:09.182124 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.182124    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="615bfc17-1ff6-41b8-9be1-7520d443a622" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:09.182227 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.182180    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="615bfc17-1ff6-41b8-9be1-7520d443a622" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:09.185134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.185120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.195523 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.195503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:09.212471 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.212440    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzq4\" (UniqueName: \"kubernetes.io/projected/615bfc17-1ff6-41b8-9be1-7520d443a622-kube-api-access-ctzq4\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:09.313669 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.313628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gslht\" (UniqueName: \"kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht\") pod \"test-curl-7612-140319604496400\" (UID: \"b335b6c5-1ff0-4d7d-985b-ba5637162c39\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.414802 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.414716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gslht\" (UniqueName: \"kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht\") pod \"test-curl-7612-140319604496400\" (UID: \"b335b6c5-1ff0-4d7d-985b-ba5637162c39\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.423930 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.423907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gslht\" (UniqueName: \"kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht\") pod \"test-curl-7612-140319604496400\" (UID: \"b335b6c5-1ff0-4d7d-985b-ba5637162c39\") " pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.494319 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.494289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.627573 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.627547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:09.629452 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:09.629426    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb335b6c5_1ff0_4d7d_985b_ba5637162c39.slice/crio-bef00039e6a721152d8f294cff06885c35fd2631a8f27cb6ad85318accdd457c WatchSource:0}: Error finding container bef00039e6a721152d8f294cff06885c35fd2631a8f27cb6ad85318accdd457c: Status 404 returned error can't find the container with id bef00039e6a721152d8f294cff06885c35fd2631a8f27cb6ad85318accdd457c
Jul 24 14:20:09.869195 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.869162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604496400" event={"ID":"b335b6c5-1ff0-4d7d-985b-ba5637162c39","Type":"ContainerStarted","Data":"bef00039e6a721152d8f294cff06885c35fd2631a8f27cb6ad85318accdd457c"}
Jul 24 14:20:09.870447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.870428    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:09.870591 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:09.870452    2579 scope.go:117] "RemoveContainer" containerID="b2e1f1c05f8d3f4b9bc225a6ec020193622f80776068423898de25ceffc4c633"
Jul 24 14:20:10.876632 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:10.876592    2579 generic.go:356] "Generic (PLEG): container finished" podID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" containerID="27fd9c3f571058e1e4342c27b53e6bb7f8f57d955b4dab4b42a233aa36829722" exitCode=0
Jul 24 14:20:10.877070 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:10.876688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604496400" event={"ID":"b335b6c5-1ff0-4d7d-985b-ba5637162c39","Type":"ContainerDied","Data":"27fd9c3f571058e1e4342c27b53e6bb7f8f57d955b4dab4b42a233aa36829722"}
Jul 24 14:20:12.019532 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.019507    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:12.047967 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.047939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319604496400_b335b6c5-1ff0-4d7d-985b-ba5637162c39/test-curl-7612-140319604496400/0.log"
Jul 24 14:20:12.071538 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.071503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:12.073280 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.073259    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319604496400"]
Jul 24 14:20:12.139055 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.138967    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gslht\" (UniqueName: \"kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht\") pod \"b335b6c5-1ff0-4d7d-985b-ba5637162c39\" (UID: \"b335b6c5-1ff0-4d7d-985b-ba5637162c39\") "
Jul 24 14:20:12.141214 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.141189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht" (OuterVolumeSpecName: "kube-api-access-gslht") pod "b335b6c5-1ff0-4d7d-985b-ba5637162c39" (UID: "b335b6c5-1ff0-4d7d-985b-ba5637162c39"). InnerVolumeSpecName "kube-api-access-gslht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:12.201901 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.201861    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:12.202313 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.202296    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:12.202392 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.202316    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:12.202447 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.202399    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" containerName="test-curl-7612-140319604496400"
Jul 24 14:20:12.205524 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.205505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:12.213889 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.213864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:12.240250 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.240220    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gslht\" (UniqueName: \"kubernetes.io/projected/b335b6c5-1ff0-4d7d-985b-ba5637162c39-kube-api-access-gslht\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:12.340690 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.340649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsvrl\" (UniqueName: \"kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl\") pod \"test-curl-7612-140319604471248\" (UID: \"89c31215-6fe6-410c-bf8c-88503877f947\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:12.441696 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.441602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsvrl\" (UniqueName: \"kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl\") pod \"test-curl-7612-140319604471248\" (UID: \"89c31215-6fe6-410c-bf8c-88503877f947\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:12.451150 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.451117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsvrl\" (UniqueName: \"kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl\") pod \"test-curl-7612-140319604471248\" (UID: \"89c31215-6fe6-410c-bf8c-88503877f947\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:12.515231 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.515189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:12.645621 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.645585    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:12.648631 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:12.648609    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c31215_6fe6_410c_bf8c_88503877f947.slice/crio-d7ae7243e95f34d510c6e3c660b3fd6f048c17aa45d3f034740f812a68aa4fe6 WatchSource:0}: Error finding container d7ae7243e95f34d510c6e3c660b3fd6f048c17aa45d3f034740f812a68aa4fe6: Status 404 returned error can't find the container with id d7ae7243e95f34d510c6e3c660b3fd6f048c17aa45d3f034740f812a68aa4fe6
Jul 24 14:20:12.811116 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.811064    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" podUID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" containerName="main" containerID="cri-o://12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b" gracePeriod=45
Jul 24 14:20:12.888537 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.888508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604471248" event={"ID":"89c31215-6fe6-410c-bf8c-88503877f947","Type":"ContainerStarted","Data":"d7ae7243e95f34d510c6e3c660b3fd6f048c17aa45d3f034740f812a68aa4fe6"}
Jul 24 14:20:12.889756 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.889736    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef00039e6a721152d8f294cff06885c35fd2631a8f27cb6ad85318accdd457c"
Jul 24 14:20:12.889872 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:12.889754    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:12.899448 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:20:12.899423    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-140319604496400\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:13.110847 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.110823    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:20:13.113437 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:20:13.113408    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-140319604496400\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" pod="opendatahub/test-curl-7612-140319604496400"
Jul 24 14:20:13.123868 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.123842    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b335b6c5-1ff0-4d7d-985b-ba5637162c39" path="/var/lib/kubelet/pods/b335b6c5-1ff0-4d7d-985b-ba5637162c39/volumes"
Jul 24 14:20:13.247830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247803    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247839    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmjld\" (UniqueName: \"kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247872    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248052 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.247981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache\") pod \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\" (UID: \"2a6d83ae-d192-4fc5-af28-0ad5666c0945\") "
Jul 24 14:20:13.248305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home" (OuterVolumeSpecName: "home") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:20:13.248305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:20:13.248305 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:20:13.248408 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248344    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.248408 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248358    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.248408 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.248367    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.250014 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.249995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm" (OuterVolumeSpecName: "dshm") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:20:13.250531 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.250514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld" (OuterVolumeSpecName: "kube-api-access-bmjld") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "kube-api-access-bmjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:13.250531 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.250521    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a6d83ae-d192-4fc5-af28-0ad5666c0945" (UID: "2a6d83ae-d192-4fc5-af28-0ad5666c0945"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:20:13.349298 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.349175    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmjld\" (UniqueName: \"kubernetes.io/projected/2a6d83ae-d192-4fc5-af28-0ad5666c0945-kube-api-access-bmjld\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.349298 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.349207    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6d83ae-d192-4fc5-af28-0ad5666c0945-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.349298 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.349221    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6d83ae-d192-4fc5-af28-0ad5666c0945-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:13.895683 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.895650    2579 generic.go:356] "Generic (PLEG): container finished" podID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" containerID="12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b" exitCode=0
Jul 24 14:20:13.895869 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.895696    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"
Jul 24 14:20:13.895869 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.895745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" event={"ID":"2a6d83ae-d192-4fc5-af28-0ad5666c0945","Type":"ContainerDied","Data":"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"}
Jul 24 14:20:13.895869 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.895794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz" event={"ID":"2a6d83ae-d192-4fc5-af28-0ad5666c0945","Type":"ContainerDied","Data":"24096de0a64bea81da3df113876f737ff18e303e4832ab86740066604eab192c"}
Jul 24 14:20:13.895869 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.895817    2579 scope.go:117] "RemoveContainer" containerID="12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"
Jul 24 14:20:13.897348 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.897320    2579 generic.go:356] "Generic (PLEG): container finished" podID="89c31215-6fe6-410c-bf8c-88503877f947" containerID="926bddf12de8c04710ff771e7d83ec17748ec99f79e10c7250558c0e8b78bb77" exitCode=0
Jul 24 14:20:13.897494 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.897377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604471248" event={"ID":"89c31215-6fe6-410c-bf8c-88503877f947","Type":"ContainerDied","Data":"926bddf12de8c04710ff771e7d83ec17748ec99f79e10c7250558c0e8b78bb77"}
Jul 24 14:20:13.904913 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.904895    2579 scope.go:117] "RemoveContainer" containerID="12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"
Jul 24 14:20:13.905164 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:20:13.905147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b\": container with ID starting with 12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b not found: ID does not exist" containerID="12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"
Jul 24 14:20:13.905205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.905174    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b"} err="failed to get container status \"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b\": rpc error: code = NotFound desc = could not find container \"12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b\": container with ID starting with 12d1f6e53601964f84058300d72b8ebc11aa2ab674b89cb97379486f3621614b not found: ID does not exist"
Jul 24 14:20:13.935136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.935104    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:20:13.941134 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:13.941114    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4cd364/rate-test-model-4cd364-kserve-65bbb889bc-djnmz"]
Jul 24 14:20:15.031761 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.031733    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.077199 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.077166    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319604471248_89c31215-6fe6-410c-bf8c-88503877f947/test-curl-7612-140319604471248/0.log"
Jul 24 14:20:15.113459 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.113430    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:15.124633 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.124600    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" path="/var/lib/kubelet/pods/2a6d83ae-d192-4fc5-af28-0ad5666c0945/volumes"
Jul 24 14:20:15.126431 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.126416    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:15.165831 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.165758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsvrl\" (UniqueName: \"kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl\") pod \"89c31215-6fe6-410c-bf8c-88503877f947\" (UID: \"89c31215-6fe6-410c-bf8c-88503877f947\") "
Jul 24 14:20:15.167923 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.167897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl" (OuterVolumeSpecName: "kube-api-access-zsvrl") pod "89c31215-6fe6-410c-bf8c-88503877f947" (UID: "89c31215-6fe6-410c-bf8c-88503877f947"). InnerVolumeSpecName "kube-api-access-zsvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:15.251663 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.251626    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:15.252080 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c31215-6fe6-410c-bf8c-88503877f947" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:15.252169 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c31215-6fe6-410c-bf8c-88503877f947" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:15.252169 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252107    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" containerName="main"
Jul 24 14:20:15.252169 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" containerName="main"
Jul 24 14:20:15.252320 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252199    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c31215-6fe6-410c-bf8c-88503877f947" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:15.252320 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.252214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6d83ae-d192-4fc5-af28-0ad5666c0945" containerName="main"
Jul 24 14:20:15.255296 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.255274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.265934 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.265910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:15.266308 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.266292    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsvrl\" (UniqueName: \"kubernetes.io/projected/89c31215-6fe6-410c-bf8c-88503877f947-kube-api-access-zsvrl\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:15.366729 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.366692    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4dv\" (UniqueName: \"kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv\") pod \"test-curl-7612-140319604471248\" (UID: \"424bad1a-77f5-4b90-8c40-7991359f9789\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.467404 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.467303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4dv\" (UniqueName: \"kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv\") pod \"test-curl-7612-140319604471248\" (UID: \"424bad1a-77f5-4b90-8c40-7991359f9789\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.478391 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.478356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4dv\" (UniqueName: \"kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv\") pod \"test-curl-7612-140319604471248\" (UID: \"424bad1a-77f5-4b90-8c40-7991359f9789\") " pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.563765 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.563718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.699513 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.699473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:15.701434 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:15.701407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod424bad1a_77f5_4b90_8c40_7991359f9789.slice/crio-0cb1f35c3292cba61b9c05c5864d445446f61c461a6954b6db9db382c209838f WatchSource:0}: Error finding container 0cb1f35c3292cba61b9c05c5864d445446f61c461a6954b6db9db382c209838f: Status 404 returned error can't find the container with id 0cb1f35c3292cba61b9c05c5864d445446f61c461a6954b6db9db382c209838f
Jul 24 14:20:15.909655 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.909628    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:15.909655 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.909657    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ae7243e95f34d510c6e3c660b3fd6f048c17aa45d3f034740f812a68aa4fe6"
Jul 24 14:20:15.910895 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.910868    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604471248" event={"ID":"424bad1a-77f5-4b90-8c40-7991359f9789","Type":"ContainerStarted","Data":"0cb1f35c3292cba61b9c05c5864d445446f61c461a6954b6db9db382c209838f"}
Jul 24 14:20:15.921842 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:15.921803    2579 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7612-140319604471248" oldPodUID="89c31215-6fe6-410c-bf8c-88503877f947" podUID="424bad1a-77f5-4b90-8c40-7991359f9789"
Jul 24 14:20:16.918015 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:16.917980    2579 generic.go:356] "Generic (PLEG): container finished" podID="424bad1a-77f5-4b90-8c40-7991359f9789" containerID="ab124fd739eb6acd556e6c18cb57cbd978b3ebe44db0452f8e9b389fb033d308" exitCode=0
Jul 24 14:20:16.918393 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:16.918051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7612-140319604471248" event={"ID":"424bad1a-77f5-4b90-8c40-7991359f9789","Type":"ContainerDied","Data":"ab124fd739eb6acd556e6c18cb57cbd978b3ebe44db0452f8e9b389fb033d308"}
Jul 24 14:20:17.124471 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:17.124435    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c31215-6fe6-410c-bf8c-88503877f947" path="/var/lib/kubelet/pods/89c31215-6fe6-410c-bf8c-88503877f947/volumes"
Jul 24 14:20:18.055518 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.055470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:18.096073 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.096047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7612-140319604471248_424bad1a-77f5-4b90-8c40-7991359f9789/test-curl-7612-140319604471248/0.log"
Jul 24 14:20:18.130452 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.130409    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:18.132102 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.132072    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7612-140319604471248"]
Jul 24 14:20:18.191024 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.190939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd4dv\" (UniqueName: \"kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv\") pod \"424bad1a-77f5-4b90-8c40-7991359f9789\" (UID: \"424bad1a-77f5-4b90-8c40-7991359f9789\") "
Jul 24 14:20:18.193344 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.193271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv" (OuterVolumeSpecName: "kube-api-access-gd4dv") pod "424bad1a-77f5-4b90-8c40-7991359f9789" (UID: "424bad1a-77f5-4b90-8c40-7991359f9789"). InnerVolumeSpecName "kube-api-access-gd4dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:20:18.291818 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.291786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4dv\" (UniqueName: \"kubernetes.io/projected/424bad1a-77f5-4b90-8c40-7991359f9789-kube-api-access-gd4dv\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:20:18.928163 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.928131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:18.928420 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:18.928169    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb1f35c3292cba61b9c05c5864d445446f61c461a6954b6db9db382c209838f"
Jul 24 14:20:18.938139 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:20:18.938112    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7612-140319604471248\" is forbidden: User \"system:node:ip-10-0-136-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-88.ec2.internal' and this object" podUID="424bad1a-77f5-4b90-8c40-7991359f9789" pod="opendatahub/test-curl-7612-140319604471248"
Jul 24 14:20:19.124856 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:19.124821    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="424bad1a-77f5-4b90-8c40-7991359f9789" path="/var/lib/kubelet/pods/424bad1a-77f5-4b90-8c40-7991359f9789/volumes"
Jul 24 14:20:21.412672 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:21.412641    2579 scope.go:117] "RemoveContainer" containerID="ba7bdaebeb12afcca931f62c4fce3665cfb54d38a1aa3df9d1c191624d1c46a2"
Jul 24 14:20:24.416838 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.416801    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:20:24.417205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.417153    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="424bad1a-77f5-4b90-8c40-7991359f9789" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:24.417205 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.417163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="424bad1a-77f5-4b90-8c40-7991359f9789" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:24.417282 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.417217    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="424bad1a-77f5-4b90-8c40-7991359f9789" containerName="test-curl-7612-140319604471248"
Jul 24 14:20:24.421899 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.421881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.424723 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.424699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-546fcb\""
Jul 24 14:20:24.439329 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.439306    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:20:24.487378 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.487349    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:20:24.490675 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.490660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.496356 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.496335    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-546fcb-dockercfg-wf8j6\""
Jul 24 14:20:24.503611 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.503589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:20:24.548193 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.548164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmnq\" (UniqueName: \"kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.548377 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.548259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.649643 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.649611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmnq\" (UniqueName: \"kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.649836 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.649658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.649836 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.649679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqcz\" (UniqueName: \"kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.649836 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.649763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.650417 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.650398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.659051 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.659033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmnq\" (UniqueName: \"kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq\") pod \"payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.732895 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.732810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:24.751183 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.751156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.751285 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.751192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqcz\" (UniqueName: \"kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.752103 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.752075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.760822 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.760797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqcz\" (UniqueName: \"kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz\") pod \"payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.800356 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.800281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:24.881393 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.881340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:20:24.883770 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:24.883745    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881eacb8_3d5a_4d42_abd9_e4ecbd52eae1.slice/crio-6b3b832493eea7691b3b079e6c2f7f153d1fe370d57e4eb712eb856eec3c6077 WatchSource:0}: Error finding container 6b3b832493eea7691b3b079e6c2f7f153d1fe370d57e4eb712eb856eec3c6077: Status 404 returned error can't find the container with id 6b3b832493eea7691b3b079e6c2f7f153d1fe370d57e4eb712eb856eec3c6077
Jul 24 14:20:24.959896 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.959859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" event={"ID":"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1","Type":"ContainerStarted","Data":"6b3b832493eea7691b3b079e6c2f7f153d1fe370d57e4eb712eb856eec3c6077"}
Jul 24 14:20:24.960903 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:24.960857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:20:24.962932 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:24.962910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf37a2b6_1ee6_437c_9363_322e8d696aba.slice/crio-dfb7c4ed698973e9853b0c7cf27c9db5c8f1b3b5f32048f5b7244bb441bc6bfa WatchSource:0}: Error finding container dfb7c4ed698973e9853b0c7cf27c9db5c8f1b3b5f32048f5b7244bb441bc6bfa: Status 404 returned error can't find the container with id dfb7c4ed698973e9853b0c7cf27c9db5c8f1b3b5f32048f5b7244bb441bc6bfa
Jul 24 14:20:25.970130 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.970088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" event={"ID":"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1","Type":"ContainerStarted","Data":"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"}
Jul 24 14:20:25.970657 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.970229    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:25.971684 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.971653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" event={"ID":"af37a2b6-1ee6-437c-9363-322e8d696aba","Type":"ContainerStarted","Data":"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"}
Jul 24 14:20:25.971830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.971699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" event={"ID":"af37a2b6-1ee6-437c-9363-322e8d696aba","Type":"ContainerStarted","Data":"dfb7c4ed698973e9853b0c7cf27c9db5c8f1b3b5f32048f5b7244bb441bc6bfa"}
Jul 24 14:20:25.971830 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.971773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:25.993382 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:25.993323    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" podStartSLOduration=1.993303374 podStartE2EDuration="1.993303374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:25.991939762 +0000 UTC m=+2765.454546640" watchObservedRunningTime="2026-07-24 14:20:25.993303374 +0000 UTC m=+2765.455910219"
Jul 24 14:20:26.015664 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:26.015613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" podStartSLOduration=2.015596634 podStartE2EDuration="2.015596634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:26.01528257 +0000 UTC m=+2765.477889432" watchObservedRunningTime="2026-07-24 14:20:26.015596634 +0000 UTC m=+2765.478203478"
Jul 24 14:20:36.978669 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:36.978632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:20:36.979089 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:36.978710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:20:49.750670 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.750633    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:20:49.755417 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.755390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.758063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.758028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-359bb9-dockercfg-4pt56\""
Jul 24 14:20:49.758187 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.758131    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-359bb9\""
Jul 24 14:20:49.763288 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.763265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.763378 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.763302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk4j\" (UniqueName: \"kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.770892 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.770868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:20:49.863888 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.863852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.864055 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.863899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk4j\" (UniqueName: \"kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.864655 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.864637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:49.878112 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:49.878084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk4j\" (UniqueName: \"kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j\") pod \"payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:50.065574 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:50.065540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:50.222658 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:50.222625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:20:50.229744 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:20:50.229706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b76b1a_7ffb_4af0_81f7_ccb0dd14d242.slice/crio-80f00f076de27cb75009f70913ef7603189e1aeaa30b9a5dbda020cb73e660fd WatchSource:0}: Error finding container 80f00f076de27cb75009f70913ef7603189e1aeaa30b9a5dbda020cb73e660fd: Status 404 returned error can't find the container with id 80f00f076de27cb75009f70913ef7603189e1aeaa30b9a5dbda020cb73e660fd
Jul 24 14:20:51.094228 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:51.094190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" event={"ID":"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242","Type":"ContainerStarted","Data":"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"}
Jul 24 14:20:51.094695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:51.094249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" event={"ID":"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242","Type":"ContainerStarted","Data":"80f00f076de27cb75009f70913ef7603189e1aeaa30b9a5dbda020cb73e660fd"}
Jul 24 14:20:51.094695 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:51.094272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:20:51.113687 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:20:51.113638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" podStartSLOduration=2.113624612 podStartE2EDuration="2.113624612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:20:51.112408449 +0000 UTC m=+2790.575015323" watchObservedRunningTime="2026-07-24 14:20:51.113624612 +0000 UTC m=+2790.576231456"
Jul 24 14:21:02.102376 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:02.102337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:21:22.418091 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.418052    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:21:22.421786 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.421766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.425738 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.425703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-546fcb\"/\"ipp-route-546fcb-kserve-self-signed-certs\""
Jul 24 14:21:22.425738 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.425727    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-546fcb\"/\"default-dockercfg-45559\""
Jul 24 14:21:22.425940 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.425737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-546fcb\"/\"openshift-service-ca.crt\""
Jul 24 14:21:22.425940 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.425727    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-546fcb\"/\"kube-root-ca.crt\""
Jul 24 14:21:22.432578 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.432557    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:21:22.555108 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxclh\" (UniqueName: \"kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.555334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.555334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.555334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.555334 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.555568 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.555372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656690 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxclh\" (UniqueName: \"kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.656874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.656863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.657219 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.657189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.657345 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.657227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.657345 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.657292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.659264 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.659244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.659414 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.659397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.666147 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.666123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxclh\" (UniqueName: \"kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh\") pod \"ipp-route-546fcb-kserve-699fdc88cf-59cnx\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") " pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.731599 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.731467    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:22.874784 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:22.874761    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:21:22.875691 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:21:22.875669    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2b3f83c_cfc6_49ae_9e07_793a3c59ceba.slice/crio-f106b889d2b79d0915c892d1cd5e380803ccf1cd279645681c9d9c378d9b6609 WatchSource:0}: Error finding container f106b889d2b79d0915c892d1cd5e380803ccf1cd279645681c9d9c378d9b6609: Status 404 returned error can't find the container with id f106b889d2b79d0915c892d1cd5e380803ccf1cd279645681c9d9c378d9b6609
Jul 24 14:21:23.257270 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:23.257229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" event={"ID":"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba","Type":"ContainerStarted","Data":"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"}
Jul 24 14:21:23.257270 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:23.257278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" event={"ID":"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba","Type":"ContainerStarted","Data":"f106b889d2b79d0915c892d1cd5e380803ccf1cd279645681c9d9c378d9b6609"}
Jul 24 14:21:32.732637 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:32.732575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:32.733065 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:32.732867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:32.745472 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:32.745441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:32.766767 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:32.766712    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" podStartSLOduration=10.766696266 podStartE2EDuration="10.766696266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:21:23.28064721 +0000 UTC m=+2822.743254055" watchObservedRunningTime="2026-07-24 14:21:32.766696266 +0000 UTC m=+2832.229303487"
Jul 24 14:21:33.320131 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:33.320100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:21:52.542063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:21:52.542027    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm_881eacb8-3d5a-4d42-abd9-e4ecbd52eae1/payload-processing/0.log"
Jul 24 14:22:00.148584 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.148539    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:00.152328 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.152305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.155907 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.155886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-269297\""
Jul 24 14:22:00.169615 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.169587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:00.179675 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.179647    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:00.183642 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.183622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.187653 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.187628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-269297-dockercfg-254cg\""
Jul 24 14:22:00.199934 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.199911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:00.304563 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.304529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.304733 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.304595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5cpk\" (UniqueName: \"kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.304733 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.304626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmbr\" (UniqueName: \"kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.304733 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.304661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.405703 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.405620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5cpk\" (UniqueName: \"kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.405703 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.405663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmbr\" (UniqueName: \"kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.405703 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.405703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.405939 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.405736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.406429 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.406411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.406472 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.406459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.417828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.417796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5cpk\" (UniqueName: \"kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.417828 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.417825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmbr\" (UniqueName: \"kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr\") pod \"payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.462648 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.462623    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:00.494145 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.494113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:00.614912 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.614884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:00.616256 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:22:00.616217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6febd690_6247_4e08_9cca_7d2088a069bf.slice/crio-a2afb397812e53e7c7e5899bb4285e2e82c4084b9e98974cfd2e646ff1ececa1 WatchSource:0}: Error finding container a2afb397812e53e7c7e5899bb4285e2e82c4084b9e98974cfd2e646ff1ececa1: Status 404 returned error can't find the container with id a2afb397812e53e7c7e5899bb4285e2e82c4084b9e98974cfd2e646ff1ececa1
Jul 24 14:22:00.656901 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:00.656804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:00.659365 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:22:00.659337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254c7031_95fd_4302_9de0_d8e8cc5886ff.slice/crio-e5e1c3b7135bc165cced7269232941c8b57c34a9b75b1e4e23ddbf92d80d74b7 WatchSource:0}: Error finding container e5e1c3b7135bc165cced7269232941c8b57c34a9b75b1e4e23ddbf92d80d74b7: Status 404 returned error can't find the container with id e5e1c3b7135bc165cced7269232941c8b57c34a9b75b1e4e23ddbf92d80d74b7
Jul 24 14:22:01.449341 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.449300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" event={"ID":"254c7031-95fd-4302-9de0-d8e8cc5886ff","Type":"ContainerStarted","Data":"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"}
Jul 24 14:22:01.449818 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.449366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" event={"ID":"254c7031-95fd-4302-9de0-d8e8cc5886ff","Type":"ContainerStarted","Data":"e5e1c3b7135bc165cced7269232941c8b57c34a9b75b1e4e23ddbf92d80d74b7"}
Jul 24 14:22:01.449818 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.449397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:01.450857 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.450834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" event={"ID":"6febd690-6247-4e08-9cca-7d2088a069bf","Type":"ContainerStarted","Data":"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"}
Jul 24 14:22:01.450987 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.450879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" event={"ID":"6febd690-6247-4e08-9cca-7d2088a069bf","Type":"ContainerStarted","Data":"a2afb397812e53e7c7e5899bb4285e2e82c4084b9e98974cfd2e646ff1ececa1"}
Jul 24 14:22:01.450987 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.450917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:01.481271 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.481222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" podStartSLOduration=1.481207951 podStartE2EDuration="1.481207951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:22:01.480151303 +0000 UTC m=+2860.942758146" watchObservedRunningTime="2026-07-24 14:22:01.481207951 +0000 UTC m=+2860.943814796"
Jul 24 14:22:01.518362 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:01.518314    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" podStartSLOduration=1.5183001059999999 podStartE2EDuration="1.518300106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:22:01.516439625 +0000 UTC m=+2860.979046468" watchObservedRunningTime="2026-07-24 14:22:01.518300106 +0000 UTC m=+2860.980906949"
Jul 24 14:22:12.458575 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:12.458467    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:12.459001 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:12.458656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:17.067904 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.067867    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"]
Jul 24 14:22:17.071603 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.071581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:17.075462 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.075445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:22:17.075928 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.075908    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:22:17.089160 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.089129    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"]
Jul 24 14:22:17.154175 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.154145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn\" (UID: \"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:17.254790 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.254753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn\" (UID: \"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:17.255444 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.255423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn\" (UID: \"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:17.381861 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.381789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:17.525941 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:17.525808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"]
Jul 24 14:22:17.528382 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:22:17.528353    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54500a2e_0d9c_43a8_81d9_1b5a40dcd2ed.slice/crio-ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb WatchSource:0}: Error finding container ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb: Status 404 returned error can't find the container with id ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb
Jul 24 14:22:18.535757 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:18.535717    2579 generic.go:356] "Generic (PLEG): container finished" podID="54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" containerID="3d7cadc5e905c1eb392e215111586e95ef7b6d54a52ec286619fab92b6824f4d" exitCode=0
Jul 24 14:22:18.536135 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:18.535807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn" event={"ID":"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed","Type":"ContainerDied","Data":"3d7cadc5e905c1eb392e215111586e95ef7b6d54a52ec286619fab92b6824f4d"}
Jul 24 14:22:18.536135 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:18.535887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn" event={"ID":"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed","Type":"ContainerStarted","Data":"ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb"}
Jul 24 14:22:19.675317 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:19.675286    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:19.773016 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:19.772983    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca\") pod \"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed\" (UID: \"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed\") "
Jul 24 14:22:19.773320 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:19.773298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" (UID: "54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:22:19.874082 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:19.874007    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:22:20.546947 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:20.546915    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"
Jul 24 14:22:20.547113 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:20.546949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn" event={"ID":"54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed","Type":"ContainerDied","Data":"ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb"}
Jul 24 14:22:20.547113 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:20.546997    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae5df75e7b6f3a9c0f2db54ae4ca301685fa2b5cad7b1c74b716f5c8aba6c4eb"
Jul 24 14:22:31.023686 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.023642    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:31.024519 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.024442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" podUID="254c7031-95fd-4302-9de0-d8e8cc5886ff" containerName="payload-processing" containerID="cri-o://466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea" gracePeriod=30
Jul 24 14:22:31.038804 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.038772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:31.039056 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.039029    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" podUID="6febd690-6247-4e08-9cca-7d2088a069bf" containerName="payload-pre-processing" containerID="cri-o://fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe" gracePeriod=30
Jul 24 14:22:31.309882 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.309859    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:31.312827 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.312806    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:31.365568 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.365536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5cpk\" (UniqueName: \"kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk\") pod \"6febd690-6247-4e08-9cca-7d2088a069bf\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") "
Jul 24 14:22:31.365730 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.365584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume\") pod \"254c7031-95fd-4302-9de0-d8e8cc5886ff\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") "
Jul 24 14:22:31.365730 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.365604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume\") pod \"6febd690-6247-4e08-9cca-7d2088a069bf\" (UID: \"6febd690-6247-4e08-9cca-7d2088a069bf\") "
Jul 24 14:22:31.365730 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.365634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlmbr\" (UniqueName: \"kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr\") pod \"254c7031-95fd-4302-9de0-d8e8cc5886ff\" (UID: \"254c7031-95fd-4302-9de0-d8e8cc5886ff\") "
Jul 24 14:22:31.366038 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.366003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "254c7031-95fd-4302-9de0-d8e8cc5886ff" (UID: "254c7031-95fd-4302-9de0-d8e8cc5886ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:22:31.366038 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.366021    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6febd690-6247-4e08-9cca-7d2088a069bf" (UID: "6febd690-6247-4e08-9cca-7d2088a069bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:22:31.367907 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.367881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr" (OuterVolumeSpecName: "kube-api-access-rlmbr") pod "254c7031-95fd-4302-9de0-d8e8cc5886ff" (UID: "254c7031-95fd-4302-9de0-d8e8cc5886ff"). InnerVolumeSpecName "kube-api-access-rlmbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:22:31.368019 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.367984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk" (OuterVolumeSpecName: "kube-api-access-x5cpk") pod "6febd690-6247-4e08-9cca-7d2088a069bf" (UID: "6febd690-6247-4e08-9cca-7d2088a069bf"). InnerVolumeSpecName "kube-api-access-x5cpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:22:31.466568 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.466529    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5cpk\" (UniqueName: \"kubernetes.io/projected/6febd690-6247-4e08-9cca-7d2088a069bf-kube-api-access-x5cpk\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.466568 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.466563    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254c7031-95fd-4302-9de0-d8e8cc5886ff-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.466568 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.466576    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6febd690-6247-4e08-9cca-7d2088a069bf-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.466802 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.466591    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlmbr\" (UniqueName: \"kubernetes.io/projected/254c7031-95fd-4302-9de0-d8e8cc5886ff-kube-api-access-rlmbr\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:22:31.602245 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.602153    2579 generic.go:356] "Generic (PLEG): container finished" podID="254c7031-95fd-4302-9de0-d8e8cc5886ff" containerID="466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea" exitCode=0
Jul 24 14:22:31.602245 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.602194    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"
Jul 24 14:22:31.602245 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.602231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" event={"ID":"254c7031-95fd-4302-9de0-d8e8cc5886ff","Type":"ContainerDied","Data":"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"}
Jul 24 14:22:31.602566 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.602281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22" event={"ID":"254c7031-95fd-4302-9de0-d8e8cc5886ff","Type":"ContainerDied","Data":"e5e1c3b7135bc165cced7269232941c8b57c34a9b75b1e4e23ddbf92d80d74b7"}
Jul 24 14:22:31.602566 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.602313    2579 scope.go:117] "RemoveContainer" containerID="466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"
Jul 24 14:22:31.603658 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.603636    2579 generic.go:356] "Generic (PLEG): container finished" podID="6febd690-6247-4e08-9cca-7d2088a069bf" containerID="fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe" exitCode=0
Jul 24 14:22:31.603771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.603702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" event={"ID":"6febd690-6247-4e08-9cca-7d2088a069bf","Type":"ContainerDied","Data":"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"}
Jul 24 14:22:31.603771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.603706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"
Jul 24 14:22:31.603771 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.603742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt" event={"ID":"6febd690-6247-4e08-9cca-7d2088a069bf","Type":"ContainerDied","Data":"a2afb397812e53e7c7e5899bb4285e2e82c4084b9e98974cfd2e646ff1ececa1"}
Jul 24 14:22:31.611399 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.611383    2579 scope.go:117] "RemoveContainer" containerID="466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"
Jul 24 14:22:31.611741 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:22:31.611716    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea\": container with ID starting with 466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea not found: ID does not exist" containerID="466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"
Jul 24 14:22:31.611858 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.611746    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea"} err="failed to get container status \"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea\": rpc error: code = NotFound desc = could not find container \"466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea\": container with ID starting with 466977a55b7130b84b3cc122da8b1e7cb9a0c857546583a3585a8e53ccd9f0ea not found: ID does not exist"
Jul 24 14:22:31.611858 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.611762    2579 scope.go:117] "RemoveContainer" containerID="fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"
Jul 24 14:22:31.620576 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.620556    2579 scope.go:117] "RemoveContainer" containerID="fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"
Jul 24 14:22:31.620822 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:22:31.620804    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe\": container with ID starting with fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe not found: ID does not exist" containerID="fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"
Jul 24 14:22:31.620896 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.620833    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe"} err="failed to get container status \"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe\": rpc error: code = NotFound desc = could not find container \"fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe\": container with ID starting with fdcf16eb4d324a46826b971f6318e47fa986eaf59c954a16e12ed7c16a661fbe not found: ID does not exist"
Jul 24 14:22:31.630156 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.630134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:31.634519 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.634501    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-269297-5c99676d58-znn22"]
Jul 24 14:22:31.645940 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.645922    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:31.656644 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:31.656617    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-269297-8648d589d8-5s9jt"]
Jul 24 14:22:32.200852 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:32.200817    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"]
Jul 24 14:22:32.206625 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:32.206591    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-269297-lh5vn"]
Jul 24 14:22:33.124995 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:33.124964    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254c7031-95fd-4302-9de0-d8e8cc5886ff" path="/var/lib/kubelet/pods/254c7031-95fd-4302-9de0-d8e8cc5886ff/volumes"
Jul 24 14:22:33.125297 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:33.125284    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" path="/var/lib/kubelet/pods/54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed/volumes"
Jul 24 14:22:33.126132 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:22:33.125773    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6febd690-6247-4e08-9cca-7d2088a069bf" path="/var/lib/kubelet/pods/6febd690-6247-4e08-9cca-7d2088a069bf/volumes"
Jul 24 14:23:04.432035 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.431991    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:23:04.432679 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.432221    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" podUID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" containerName="payload-processing" containerID="cri-o://acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac" gracePeriod=30
Jul 24 14:23:04.439316 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.439288    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:23:04.439627 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.439580    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" podUID="af37a2b6-1ee6-437c-9363-322e8d696aba" containerName="payload-pre-processing" containerID="cri-o://1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8" gracePeriod=30
Jul 24 14:23:04.716054 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.716032    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:23:04.718977 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.718957    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:23:04.768707 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.768675    2579 generic.go:356] "Generic (PLEG): container finished" podID="af37a2b6-1ee6-437c-9363-322e8d696aba" containerID="1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8" exitCode=0
Jul 24 14:23:04.768904 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.768727    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"
Jul 24 14:23:04.768904 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.768768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" event={"ID":"af37a2b6-1ee6-437c-9363-322e8d696aba","Type":"ContainerDied","Data":"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"}
Jul 24 14:23:04.768904 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.768832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z" event={"ID":"af37a2b6-1ee6-437c-9363-322e8d696aba","Type":"ContainerDied","Data":"dfb7c4ed698973e9853b0c7cf27c9db5c8f1b3b5f32048f5b7244bb441bc6bfa"}
Jul 24 14:23:04.768904 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.768863    2579 scope.go:117] "RemoveContainer" containerID="1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"
Jul 24 14:23:04.770234 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.770213    2579 generic.go:356] "Generic (PLEG): container finished" podID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" containerID="acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac" exitCode=0
Jul 24 14:23:04.770354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.770275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" event={"ID":"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1","Type":"ContainerDied","Data":"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"}
Jul 24 14:23:04.770354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.770320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm" event={"ID":"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1","Type":"ContainerDied","Data":"6b3b832493eea7691b3b079e6c2f7f153d1fe370d57e4eb712eb856eec3c6077"}
Jul 24 14:23:04.770354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.770278    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"
Jul 24 14:23:04.778737 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.778716    2579 scope.go:117] "RemoveContainer" containerID="1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"
Jul 24 14:23:04.779015 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:23:04.778997    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8\": container with ID starting with 1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8 not found: ID does not exist" containerID="1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"
Jul 24 14:23:04.779080 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.779025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8"} err="failed to get container status \"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8\": rpc error: code = NotFound desc = could not find container \"1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8\": container with ID starting with 1d72758acc1fec855a80f18649e64b571fff09c2531efcb193663c8f34a9e6a8 not found: ID does not exist"
Jul 24 14:23:04.779080 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.779042    2579 scope.go:117] "RemoveContainer" containerID="acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"
Jul 24 14:23:04.786911 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.786890    2579 scope.go:117] "RemoveContainer" containerID="acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"
Jul 24 14:23:04.787170 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:23:04.787149    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac\": container with ID starting with acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac not found: ID does not exist" containerID="acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"
Jul 24 14:23:04.787263 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.787178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac"} err="failed to get container status \"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac\": rpc error: code = NotFound desc = could not find container \"acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac\": container with ID starting with acecb6155f8645bb10e06faeca3e4f0fb8cc92db7eecb3350c24c541ef53bcac not found: ID does not exist"
Jul 24 14:23:04.867136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvqcz\" (UniqueName: \"kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz\") pod \"af37a2b6-1ee6-437c-9363-322e8d696aba\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") "
Jul 24 14:23:04.867297 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume\") pod \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") "
Jul 24 14:23:04.867297 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867216    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume\") pod \"af37a2b6-1ee6-437c-9363-322e8d696aba\" (UID: \"af37a2b6-1ee6-437c-9363-322e8d696aba\") "
Jul 24 14:23:04.867297 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghmnq\" (UniqueName: \"kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq\") pod \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\" (UID: \"881eacb8-3d5a-4d42-abd9-e4ecbd52eae1\") "
Jul 24 14:23:04.867634 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af37a2b6-1ee6-437c-9363-322e8d696aba" (UID: "af37a2b6-1ee6-437c-9363-322e8d696aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:23:04.867734 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.867630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" (UID: "881eacb8-3d5a-4d42-abd9-e4ecbd52eae1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:23:04.869413 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.869390    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq" (OuterVolumeSpecName: "kube-api-access-ghmnq") pod "881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" (UID: "881eacb8-3d5a-4d42-abd9-e4ecbd52eae1"). InnerVolumeSpecName "kube-api-access-ghmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:23:04.869556 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.869523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz" (OuterVolumeSpecName: "kube-api-access-bvqcz") pod "af37a2b6-1ee6-437c-9363-322e8d696aba" (UID: "af37a2b6-1ee6-437c-9363-322e8d696aba"). InnerVolumeSpecName "kube-api-access-bvqcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:23:04.968262 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.968179    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvqcz\" (UniqueName: \"kubernetes.io/projected/af37a2b6-1ee6-437c-9363-322e8d696aba-kube-api-access-bvqcz\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:04.968262 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.968217    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:04.968262 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.968227    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af37a2b6-1ee6-437c-9363-322e8d696aba-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:04.968262 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:04.968235    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmnq\" (UniqueName: \"kubernetes.io/projected/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1-kube-api-access-ghmnq\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:05.094660 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.094628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:23:05.099677 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.099654    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-546fcb-667f699768-9t88z"]
Jul 24 14:23:05.110486 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.110454    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:23:05.114812 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.114791    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-546fcb-5686d7d4bb-n9dcm"]
Jul 24 14:23:05.124797 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.124777    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" path="/var/lib/kubelet/pods/881eacb8-3d5a-4d42-abd9-e4ecbd52eae1/volumes"
Jul 24 14:23:05.125118 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:05.125106    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af37a2b6-1ee6-437c-9363-322e8d696aba" path="/var/lib/kubelet/pods/af37a2b6-1ee6-437c-9363-322e8d696aba/volumes"
Jul 24 14:23:14.525940 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:14.525906    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:23:29.545546 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.545467    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" podUID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" containerName="main" containerID="cri-o://88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2" gracePeriod=45
Jul 24 14:23:29.798812 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.798752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:23:29.892687 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxclh\" (UniqueName: \"kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.892875 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.892875 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.892875 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.892875 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.892875 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.892830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache\") pod \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\" (UID: \"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba\") "
Jul 24 14:23:29.893136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.893006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home" (OuterVolumeSpecName: "home") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:23:29.893136 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.893031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:23:29.893257 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.893203    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-home\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:29.893257 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.893225    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tmp-dir\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:29.893257 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.893235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:23:29.894033 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.894005    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" containerID="88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2" exitCode=0
Jul 24 14:23:29.894159 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.894059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"
Jul 24 14:23:29.894159 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.894135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" event={"ID":"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba","Type":"ContainerDied","Data":"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"}
Jul 24 14:23:29.894282 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.894183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx" event={"ID":"d2b3f83c-cfc6-49ae-9e07-793a3c59ceba","Type":"ContainerDied","Data":"f106b889d2b79d0915c892d1cd5e380803ccf1cd279645681c9d9c378d9b6609"}
Jul 24 14:23:29.894282 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.894208    2579 scope.go:117] "RemoveContainer" containerID="88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"
Jul 24 14:23:29.895517 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.895496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm" (OuterVolumeSpecName: "dshm") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 14:23:29.895613 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.895566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh" (OuterVolumeSpecName: "kube-api-access-sxclh") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "kube-api-access-sxclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:23:29.895613 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.895566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" (UID: "d2b3f83c-cfc6-49ae-9e07-793a3c59ceba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 14:23:29.912743 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.912722    2579 scope.go:117] "RemoveContainer" containerID="88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"
Jul 24 14:23:29.913017 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:23:29.913000    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2\": container with ID starting with 88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2 not found: ID does not exist" containerID="88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"
Jul 24 14:23:29.913084 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.913026    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2"} err="failed to get container status \"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2\": rpc error: code = NotFound desc = could not find container \"88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2\": container with ID starting with 88d21ebd73e1a538db046ccda3a0d1e69aab44e1410e73c5d0757034afbc57c2 not found: ID does not exist"
Jul 24 14:23:29.995573 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.995546    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-tls-certs\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:29.995573 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.995570    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-model-cache\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:29.995573 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.995579    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxclh\" (UniqueName: \"kubernetes.io/projected/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-kube-api-access-sxclh\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:29.995768 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:29.995586    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba-dshm\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:23:30.216155 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:30.216125    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:23:30.221190 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:30.221170    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-546fcb/ipp-route-546fcb-kserve-699fdc88cf-59cnx"]
Jul 24 14:23:31.124787 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:23:31.124750    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" path="/var/lib/kubelet/pods/d2b3f83c-cfc6-49ae-9e07-793a3c59ceba/volumes"
Jul 24 14:24:00.336148 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336111    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"]
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6febd690-6247-4e08-9cca-7d2088a069bf" containerName="payload-pre-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336509    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6febd690-6247-4e08-9cca-7d2088a069bf" containerName="payload-pre-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336523    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" containerName="revoke-keys"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336531    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" containerName="revoke-keys"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af37a2b6-1ee6-437c-9363-322e8d696aba" containerName="payload-pre-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336546    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af37a2b6-1ee6-437c-9363-322e8d696aba" containerName="payload-pre-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336562    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" containerName="main"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336567    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" containerName="main"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336575    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" containerName="payload-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336580    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" containerName="payload-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336588    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254c7031-95fd-4302-9de0-d8e8cc5886ff" containerName="payload-processing"
Jul 24 14:24:00.336596 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336593    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="254c7031-95fd-4302-9de0-d8e8cc5886ff" containerName="payload-processing"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336645    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b3f83c-cfc6-49ae-9e07-793a3c59ceba" containerName="main"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336654    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54500a2e-0d9c-43a8-81d9-1b5a40dcd2ed" containerName="revoke-keys"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336660    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="881eacb8-3d5a-4d42-abd9-e4ecbd52eae1" containerName="payload-processing"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336665    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6febd690-6247-4e08-9cca-7d2088a069bf" containerName="payload-pre-processing"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336671    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af37a2b6-1ee6-437c-9363-322e8d696aba" containerName="payload-pre-processing"
Jul 24 14:24:00.336966 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.336677    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="254c7031-95fd-4302-9de0-d8e8cc5886ff" containerName="payload-processing"
Jul 24 14:24:00.339504 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.339488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:00.342179 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.342161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:24:00.342653 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.342639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:24:00.354293 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.354268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"]
Jul 24 14:24:00.361438 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.361408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr\" (UID: \"75926c15-fe14-4ca0-ab0f-9959a3dca317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:00.462012 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.461979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr\" (UID: \"75926c15-fe14-4ca0-ab0f-9959a3dca317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:00.462644 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.462622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr\" (UID: \"75926c15-fe14-4ca0-ab0f-9959a3dca317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:00.650189 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.650102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:00.789739 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:00.789712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"]
Jul 24 14:24:00.793665 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:24:00.793631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75926c15_fe14_4ca0_ab0f_9959a3dca317.slice/crio-64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130 WatchSource:0}: Error finding container 64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130: Status 404 returned error can't find the container with id 64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130
Jul 24 14:24:01.051714 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:01.051674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr" event={"ID":"75926c15-fe14-4ca0-ab0f-9959a3dca317","Type":"ContainerStarted","Data":"64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130"}
Jul 24 14:24:02.057972 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:02.057934    2579 generic.go:356] "Generic (PLEG): container finished" podID="75926c15-fe14-4ca0-ab0f-9959a3dca317" containerID="6ef1b90a91966b4024fddcb14602eeeb33dc1f024bbf936beefa1f8e4e55e9d4" exitCode=0
Jul 24 14:24:02.058411 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:02.058022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr" event={"ID":"75926c15-fe14-4ca0-ab0f-9959a3dca317","Type":"ContainerDied","Data":"6ef1b90a91966b4024fddcb14602eeeb33dc1f024bbf936beefa1f8e4e55e9d4"}
Jul 24 14:24:03.192319 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:03.192294    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:03.287354 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:03.287321    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca\") pod \"75926c15-fe14-4ca0-ab0f-9959a3dca317\" (UID: \"75926c15-fe14-4ca0-ab0f-9959a3dca317\") "
Jul 24 14:24:03.287720 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:03.287696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75926c15-fe14-4ca0-ab0f-9959a3dca317" (UID: "75926c15-fe14-4ca0-ab0f-9959a3dca317"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:03.388848 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:03.388756    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75926c15-fe14-4ca0-ab0f-9959a3dca317-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:04.068274 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:04.068240    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"
Jul 24 14:24:04.068452 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:04.068271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr" event={"ID":"75926c15-fe14-4ca0-ab0f-9959a3dca317","Type":"ContainerDied","Data":"64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130"}
Jul 24 14:24:04.068452 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:04.068316    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64bd0f57b3427b13b76a2af45a3dee4fabd19b337d6e487f5fab80245819e130"
Jul 24 14:24:11.786661 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:11.786630    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:24:11.787104 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:11.786911    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" podUID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" containerName="payload-pre-processing" containerID="cri-o://3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f" gracePeriod=30
Jul 24 14:24:12.044164 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.044098    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:24:12.107376 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.107339    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" containerID="3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f" exitCode=0
Jul 24 14:24:12.107588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.107381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"
Jul 24 14:24:12.107588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.107386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" event={"ID":"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242","Type":"ContainerDied","Data":"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"}
Jul 24 14:24:12.107588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.107530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p" event={"ID":"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242","Type":"ContainerDied","Data":"80f00f076de27cb75009f70913ef7603189e1aeaa30b9a5dbda020cb73e660fd"}
Jul 24 14:24:12.107588 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.107566    2579 scope.go:117] "RemoveContainer" containerID="3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"
Jul 24 14:24:12.116841 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.116818    2579 scope.go:117] "RemoveContainer" containerID="3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"
Jul 24 14:24:12.117066 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:24:12.117045    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f\": container with ID starting with 3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f not found: ID does not exist" containerID="3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"
Jul 24 14:24:12.117167 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.117073    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f"} err="failed to get container status \"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f\": rpc error: code = NotFound desc = could not find container \"3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f\": container with ID starting with 3bfde355af63f82034f486b658738d4309a2bc641ac9f5bba5a27996d5aa363f not found: ID does not exist"
Jul 24 14:24:12.164102 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.164074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume\") pod \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") "
Jul 24 14:24:12.164266 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.164112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gk4j\" (UniqueName: \"kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j\") pod \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\" (UID: \"b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242\") "
Jul 24 14:24:12.164433 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.164410    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" (UID: "b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:12.166386 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.166361    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j" (OuterVolumeSpecName: "kube-api-access-9gk4j") pod "b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" (UID: "b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242"). InnerVolumeSpecName "kube-api-access-9gk4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:12.265665 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.265636    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.265665 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.265663    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk4j\" (UniqueName: \"kubernetes.io/projected/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242-kube-api-access-9gk4j\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:12.430786 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.430758    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:24:12.438081 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:12.438058    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-359bb9-5b956f994d-x4p9p"]
Jul 24 14:24:13.124529 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:13.124496    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" path="/var/lib/kubelet/pods/b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242/volumes"
Jul 24 14:24:15.580796 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:15.580754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"]
Jul 24 14:24:15.582267 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:15.582243    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-359bb9-m7dkr"]
Jul 24 14:24:17.124045 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:17.124011    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75926c15-fe14-4ca0-ab0f-9959a3dca317" path="/var/lib/kubelet/pods/75926c15-fe14-4ca0-ab0f-9959a3dca317/volumes"
Jul 24 14:24:21.199971 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:21.199862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:24:21.208873 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:21.208854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbhmq_0b1f159e-12a1-418a-97d6-8de1de5fa880/ovn-acl-logging/0.log"
Jul 24 14:24:34.023521 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.023464    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"]
Jul 24 14:24:34.025790 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.025763    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cd364-29748375-zc692"]
Jul 24 14:24:34.078800 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.078764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:24:34.079005 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.078962    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" podUID="02023676-3984-45d1-bc98-afa609b9e529" containerName="payload-processing" containerID="cri-o://2c5b66732f662be8b071c1e24898b9a6a000f8f202faf3401a7616ed0a0acb60" gracePeriod=30
Jul 24 14:24:34.085837 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.085497    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:24:34.085837 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.085723    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" containerName="payload-pre-processing" containerID="cri-o://c8ccd7894d0fb081ac7206a30feb7b7fc2c21ce6cf66b298a996a773618c26b6" gracePeriod=30
Jul 24 14:24:34.213163 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.213127    2579 generic.go:356] "Generic (PLEG): container finished" podID="5d067f51-7d11-458e-865c-1f24163acf5e" containerID="c8ccd7894d0fb081ac7206a30feb7b7fc2c21ce6cf66b298a996a773618c26b6" exitCode=0
Jul 24 14:24:34.213333 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.213205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" event={"ID":"5d067f51-7d11-458e-865c-1f24163acf5e","Type":"ContainerDied","Data":"c8ccd7894d0fb081ac7206a30feb7b7fc2c21ce6cf66b298a996a773618c26b6"}
Jul 24 14:24:34.215225 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.215200    2579 generic.go:356] "Generic (PLEG): container finished" podID="02023676-3984-45d1-bc98-afa609b9e529" containerID="2c5b66732f662be8b071c1e24898b9a6a000f8f202faf3401a7616ed0a0acb60" exitCode=0
Jul 24 14:24:34.215340 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.215230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" event={"ID":"02023676-3984-45d1-bc98-afa609b9e529","Type":"ContainerDied","Data":"2c5b66732f662be8b071c1e24898b9a6a000f8f202faf3401a7616ed0a0acb60"}
Jul 24 14:24:34.347385 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.347364    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:24:34.357585 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.357565    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:24:34.458316 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume\") pod \"02023676-3984-45d1-bc98-afa609b9e529\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") "
Jul 24 14:24:34.458549 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rqbw\" (UniqueName: \"kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw\") pod \"5d067f51-7d11-458e-865c-1f24163acf5e\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") "
Jul 24 14:24:34.458549 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume\") pod \"5d067f51-7d11-458e-865c-1f24163acf5e\" (UID: \"5d067f51-7d11-458e-865c-1f24163acf5e\") "
Jul 24 14:24:34.458549 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2flb\" (UniqueName: \"kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb\") pod \"02023676-3984-45d1-bc98-afa609b9e529\" (UID: \"02023676-3984-45d1-bc98-afa609b9e529\") "
Jul 24 14:24:34.458721 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02023676-3984-45d1-bc98-afa609b9e529" (UID: "02023676-3984-45d1-bc98-afa609b9e529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:34.458810 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458791    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02023676-3984-45d1-bc98-afa609b9e529-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.458874 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.458804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d067f51-7d11-458e-865c-1f24163acf5e" (UID: "5d067f51-7d11-458e-865c-1f24163acf5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:34.460648 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.460626    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb" (OuterVolumeSpecName: "kube-api-access-x2flb") pod "02023676-3984-45d1-bc98-afa609b9e529" (UID: "02023676-3984-45d1-bc98-afa609b9e529"). InnerVolumeSpecName "kube-api-access-x2flb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:34.460648 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.460632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw" (OuterVolumeSpecName: "kube-api-access-7rqbw") pod "5d067f51-7d11-458e-865c-1f24163acf5e" (UID: "5d067f51-7d11-458e-865c-1f24163acf5e"). InnerVolumeSpecName "kube-api-access-7rqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:34.559885 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.559806    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d067f51-7d11-458e-865c-1f24163acf5e-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.559885 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.559832    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2flb\" (UniqueName: \"kubernetes.io/projected/02023676-3984-45d1-bc98-afa609b9e529-kube-api-access-x2flb\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:34.559885 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:34.559843    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rqbw\" (UniqueName: \"kubernetes.io/projected/5d067f51-7d11-458e-865c-1f24163acf5e-kube-api-access-7rqbw\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:35.124191 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.124162    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b14685f0-8d12-446c-9952-4cc8cfba7184" path="/var/lib/kubelet/pods/b14685f0-8d12-446c-9952-4cc8cfba7184/volumes"
Jul 24 14:24:35.221591 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.221565    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"
Jul 24 14:24:35.221782 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.221602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz" event={"ID":"5d067f51-7d11-458e-865c-1f24163acf5e","Type":"ContainerDied","Data":"5e9f983fcbc71530cbbdd73c4ba6519c5b15fec07afc6bab45bd4e28403d5e3a"}
Jul 24 14:24:35.221782 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.221662    2579 scope.go:117] "RemoveContainer" containerID="c8ccd7894d0fb081ac7206a30feb7b7fc2c21ce6cf66b298a996a773618c26b6"
Jul 24 14:24:35.223007 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.222957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg" event={"ID":"02023676-3984-45d1-bc98-afa609b9e529","Type":"ContainerDied","Data":"66e345ecc460277fa7317fad690c76a122b9d3e171f9cc1953bc7f138529112a"}
Jul 24 14:24:35.223007 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.222989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"
Jul 24 14:24:35.230594 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.230521    2579 scope.go:117] "RemoveContainer" containerID="2c5b66732f662be8b071c1e24898b9a6a000f8f202faf3401a7616ed0a0acb60"
Jul 24 14:24:35.245441 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.245419    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:24:35.246932 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.246911    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-4cd364-7bdbf4d58d-xccdz"]
Jul 24 14:24:35.264844 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.264824    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:24:35.272237 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:35.272218    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cd364-df86b59f4-8f6mg"]
Jul 24 14:24:37.124343 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:37.124312    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02023676-3984-45d1-bc98-afa609b9e529" path="/var/lib/kubelet/pods/02023676-3984-45d1-bc98-afa609b9e529/volumes"
Jul 24 14:24:37.124731 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:37.124673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" path="/var/lib/kubelet/pods/5d067f51-7d11-458e-865c-1f24163acf5e/volumes"
Jul 24 14:24:46.685710 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.685675    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"]
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686018    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" containerName="payload-pre-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" containerName="payload-pre-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686052    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02023676-3984-45d1-bc98-afa609b9e529" containerName="payload-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686057    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02023676-3984-45d1-bc98-afa609b9e529" containerName="payload-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686067    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" containerName="payload-pre-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686072    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" containerName="payload-pre-processing"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686079    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75926c15-fe14-4ca0-ab0f-9959a3dca317" containerName="revoke-keys"
Jul 24 14:24:46.686125 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="75926c15-fe14-4ca0-ab0f-9959a3dca317" containerName="revoke-keys"
Jul 24 14:24:46.686381 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686139    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b76b1a-7ffb-4af0-81f7-ccb0dd14d242" containerName="payload-pre-processing"
Jul 24 14:24:46.686381 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686145    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d067f51-7d11-458e-865c-1f24163acf5e" containerName="payload-pre-processing"
Jul 24 14:24:46.686381 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686152    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="75926c15-fe14-4ca0-ab0f-9959a3dca317" containerName="revoke-keys"
Jul 24 14:24:46.686381 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.686158    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02023676-3984-45d1-bc98-afa609b9e529" containerName="payload-processing"
Jul 24 14:24:46.690592 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.690571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:46.693888 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.693869    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 14:24:46.693975 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.693869    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x4stw\""
Jul 24 14:24:46.705837 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.705815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"]
Jul 24 14:24:46.858370 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.858318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-525009-mmx6m\" (UID: \"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:46.959038 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.958933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-525009-mmx6m\" (UID: \"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:46.959603 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:46.959584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-525009-mmx6m\" (UID: \"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:47.000612 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:47.000567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:47.143575 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:47.143540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"]
Jul 24 14:24:47.147050 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:24:47.147022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd55a5e_cb29_41b6_9727_22d5d8bf25f2.slice/crio-3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd WatchSource:0}: Error finding container 3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd: Status 404 returned error can't find the container with id 3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd
Jul 24 14:24:47.282702 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:47.282669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m" event={"ID":"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2","Type":"ContainerStarted","Data":"3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd"}
Jul 24 14:24:48.289392 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:48.289361    2579 generic.go:356] "Generic (PLEG): container finished" podID="dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" containerID="4f998551642114fde58d893769a226e11089a6c227cb934692c0877fdeb5d3b6" exitCode=0
Jul 24 14:24:48.289770 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:48.289438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m" event={"ID":"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2","Type":"ContainerDied","Data":"4f998551642114fde58d893769a226e11089a6c227cb934692c0877fdeb5d3b6"}
Jul 24 14:24:49.427427 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:49.427397    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:49.478915 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:49.478886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca\") pod \"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2\" (UID: \"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2\") "
Jul 24 14:24:49.479286 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:49.479263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" (UID: "dcd55a5e-cb29-41b6-9727-22d5d8bf25f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:49.579576 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:49.579501    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2-maas-api-service-ca\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:50.300384 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:50.300355    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"
Jul 24 14:24:50.300610 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:50.300356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m" event={"ID":"dcd55a5e-cb29-41b6-9727-22d5d8bf25f2","Type":"ContainerDied","Data":"3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd"}
Jul 24 14:24:50.300610 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:50.300498    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3add6b756dded76f27dfa8ac30d14dca8121803407dbf9fbbd07be008de8a7cd"
Jul 24 14:24:57.745229 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:57.745193    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:24:57.745802 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:57.745455    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" podUID="8816c977-47be-4d3e-b536-e01e23fb26fb" containerName="payload-pre-processing" containerID="cri-o://2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e" gracePeriod=30
Jul 24 14:24:57.983023 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:57.983000    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:24:58.037542 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.037507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume\") pod \"8816c977-47be-4d3e-b536-e01e23fb26fb\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") "
Jul 24 14:24:58.037687 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.037566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt879\" (UniqueName: \"kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879\") pod \"8816c977-47be-4d3e-b536-e01e23fb26fb\" (UID: \"8816c977-47be-4d3e-b536-e01e23fb26fb\") "
Jul 24 14:24:58.037844 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.037812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8816c977-47be-4d3e-b536-e01e23fb26fb" (UID: "8816c977-47be-4d3e-b536-e01e23fb26fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 14:24:58.037949 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.037908    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8816c977-47be-4d3e-b536-e01e23fb26fb-plugins-config-volume\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.039715 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.039691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879" (OuterVolumeSpecName: "kube-api-access-dt879") pod "8816c977-47be-4d3e-b536-e01e23fb26fb" (UID: "8816c977-47be-4d3e-b536-e01e23fb26fb"). InnerVolumeSpecName "kube-api-access-dt879". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 14:24:58.138312 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.138270    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt879\" (UniqueName: \"kubernetes.io/projected/8816c977-47be-4d3e-b536-e01e23fb26fb-kube-api-access-dt879\") on node \"ip-10-0-136-88.ec2.internal\" DevicePath \"\""
Jul 24 14:24:58.342423 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.342335    2579 generic.go:356] "Generic (PLEG): container finished" podID="8816c977-47be-4d3e-b536-e01e23fb26fb" containerID="2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e" exitCode=0
Jul 24 14:24:58.342423 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.342383    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"
Jul 24 14:24:58.342633 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.342386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" event={"ID":"8816c977-47be-4d3e-b536-e01e23fb26fb","Type":"ContainerDied","Data":"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"}
Jul 24 14:24:58.342633 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.342542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m" event={"ID":"8816c977-47be-4d3e-b536-e01e23fb26fb","Type":"ContainerDied","Data":"6bd9ae777421f746b73d811238302e17b0122e2a82e18b982a3b1583ad4f35dd"}
Jul 24 14:24:58.342633 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.342575    2579 scope.go:117] "RemoveContainer" containerID="2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"
Jul 24 14:24:58.351834 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.351811    2579 scope.go:117] "RemoveContainer" containerID="2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"
Jul 24 14:24:58.352116 ip-10-0-136-88 kubenswrapper[2579]: E0724 14:24:58.352096    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e\": container with ID starting with 2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e not found: ID does not exist" containerID="2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"
Jul 24 14:24:58.352182 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.352131    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e"} err="failed to get container status \"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e\": rpc error: code = NotFound desc = could not find container \"2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e\": container with ID starting with 2a9c5eefe4daa5cc9cf30a61c34c54e835c5180242fcff38a6ef9a4f16ba784e not found: ID does not exist"
Jul 24 14:24:58.369786 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.369757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:24:58.374504 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:58.374458    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-525009-7d577fb5c6-npb5m"]
Jul 24 14:24:59.124544 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:24:59.124513    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8816c977-47be-4d3e-b536-e01e23fb26fb" path="/var/lib/kubelet/pods/8816c977-47be-4d3e-b536-e01e23fb26fb/volumes"
Jul 24 14:25:01.804858 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:01.804831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"]
Jul 24 14:25:01.809970 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:01.809946    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-525009-mmx6m"]
Jul 24 14:25:03.123976 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:03.123941    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" path="/var/lib/kubelet/pods/dcd55a5e-cb29-41b6-9727-22d5d8bf25f2/volumes"
Jul 24 14:25:09.762415 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:09.762379    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f757f4d8-qfq8x_16e70d91-7dab-4d62-bdf2-acc562e92b7b/authorino/0.log"
Jul 24 14:25:16.022576 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:16.022504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-bvcb6_5a067aa3-05ca-4af7-88a5-89e5f04ad91e/copy-manifests/0.log"
Jul 24 14:25:16.032595 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:16.032564    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-bvcb6_5a067aa3-05ca-4af7-88a5-89e5f04ad91e/manager/0.log"
Jul 24 14:25:16.868030 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:16.867998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748375-n6wgv_1a8acb7a-d03a-4ce8-8883-b33f3e5e2db9/cleanup/0.log"
Jul 24 14:25:18.367438 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:18.367405    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f757f4d8-qfq8x_16e70d91-7dab-4d62-bdf2-acc562e92b7b/authorino/0.log"
Jul 24 14:25:18.725196 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:18.725110    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b8ksv_40c1d197-bad6-4963-a86a-30f31a209818/kuadrant-console-plugin/0.log"
Jul 24 14:25:19.243133 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:19.243082    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nwdnl_75b2e2b1-7cce-4e32-99ab-a203716b8bab/manager/0.log"
Jul 24 14:25:19.991197 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:19.991166    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85dc78bf9-jflxb_feb7105e-55dc-4bf2-a756-22626d6c0e72/kube-auth-proxy/0.log"
Jul 24 14:25:20.942766 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:20.942727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gfchr_124a8b34-146c-430b-ab44-ef63952d1168/main/0.log"
Jul 24 14:25:21.064680 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:21.064653    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-547k5_2d05bbeb-fc73-41f2-b926-657e5450d623/main/0.log"
Jul 24 14:25:21.505690 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:21.505662    2579 scope.go:117] "RemoveContainer" containerID="378d76a8def1190f25fa9100e6274eeb1360f63352d2eb8afa2fb41f2ec045fd"
Jul 24 14:25:26.698033 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698000    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hh2p5/must-gather-twhf6"]
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698308    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" containerName="revoke-keys"
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698319    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" containerName="revoke-keys"
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698342    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8816c977-47be-4d3e-b536-e01e23fb26fb" containerName="payload-pre-processing"
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698348    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8816c977-47be-4d3e-b536-e01e23fb26fb" containerName="payload-pre-processing"
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698402    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8816c977-47be-4d3e-b536-e01e23fb26fb" containerName="payload-pre-processing"
Jul 24 14:25:26.698624 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.698411    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd55a5e-cb29-41b6-9727-22d5d8bf25f2" containerName="revoke-keys"
Jul 24 14:25:26.702841 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.702820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.709758 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.709737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hh2p5\"/\"kube-root-ca.crt\""
Jul 24 14:25:26.709842 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.709737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hh2p5\"/\"openshift-service-ca.crt\""
Jul 24 14:25:26.714999 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.714983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hh2p5\"/\"default-dockercfg-gvc2t\""
Jul 24 14:25:26.741333 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.741302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/must-gather-twhf6"]
Jul 24 14:25:26.760961 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.760935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6ls\" (UniqueName: \"kubernetes.io/projected/35da7bf2-fa12-433a-926e-e57708616c0c-kube-api-access-zt6ls\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.761091 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.760981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35da7bf2-fa12-433a-926e-e57708616c0c-must-gather-output\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.862425 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.862389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6ls\" (UniqueName: \"kubernetes.io/projected/35da7bf2-fa12-433a-926e-e57708616c0c-kube-api-access-zt6ls\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.862614 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.862436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35da7bf2-fa12-433a-926e-e57708616c0c-must-gather-output\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.862756 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.862742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/35da7bf2-fa12-433a-926e-e57708616c0c-must-gather-output\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:26.872844 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:26.872814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6ls\" (UniqueName: \"kubernetes.io/projected/35da7bf2-fa12-433a-926e-e57708616c0c-kube-api-access-zt6ls\") pod \"must-gather-twhf6\" (UID: \"35da7bf2-fa12-433a-926e-e57708616c0c\") " pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:27.012328 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:27.012292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/must-gather-twhf6"
Jul 24 14:25:27.172950 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:27.172917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/must-gather-twhf6"]
Jul 24 14:25:27.175258 ip-10-0-136-88 kubenswrapper[2579]: W0724 14:25:27.175230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35da7bf2_fa12_433a_926e_e57708616c0c.slice/crio-2350d31fb3219b104086842489aa486f7cd41b8753eb55928dc60d059e97f7af WatchSource:0}: Error finding container 2350d31fb3219b104086842489aa486f7cd41b8753eb55928dc60d059e97f7af: Status 404 returned error can't find the container with id 2350d31fb3219b104086842489aa486f7cd41b8753eb55928dc60d059e97f7af
Jul 24 14:25:27.177133 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:27.177115    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 14:25:27.488081 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:27.487991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/must-gather-twhf6" event={"ID":"35da7bf2-fa12-433a-926e-e57708616c0c","Type":"ContainerStarted","Data":"2350d31fb3219b104086842489aa486f7cd41b8753eb55928dc60d059e97f7af"}
Jul 24 14:25:28.496772 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:28.496145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/must-gather-twhf6" event={"ID":"35da7bf2-fa12-433a-926e-e57708616c0c","Type":"ContainerStarted","Data":"83c34ba55191b44d78d23af0cb57c083121a067e03d8f9e7138a12036cf07b5a"}
Jul 24 14:25:28.496772 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:28.496200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/must-gather-twhf6" event={"ID":"35da7bf2-fa12-433a-926e-e57708616c0c","Type":"ContainerStarted","Data":"fa0f9e072b4698fe93622479244570142c2ce5c22af3625fdebbb95d6855c702"}
Jul 24 14:25:28.519063 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:28.519011    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hh2p5/must-gather-twhf6" podStartSLOduration=1.772091526 podStartE2EDuration="2.518995392s" totalImagesPullingTime="746.903866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:28.51727014 +0000 UTC m=+3067.979876984" watchObservedRunningTime="2026-07-24 14:25:28.518995392 +0000 UTC m=+3067.981602236"
Jul 24 14:25:29.593224 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:29.593193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j8frr_c449e0c2-6d25-4452-b9ba-0e3a08c26fdc/global-pull-secret-syncer/0.log"
Jul 24 14:25:29.763843 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:29.763812    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s6nth_d681c4f4-490d-4200-8220-72d44766fa11/konnectivity-agent/0.log"
Jul 24 14:25:29.812155 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:29.812125    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-88.ec2.internal_86bdcc87ec35dd1e444c5a9b19ae75f0/haproxy/0.log"
Jul 24 14:25:34.349331 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:34.349293    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65f757f4d8-qfq8x_16e70d91-7dab-4d62-bdf2-acc562e92b7b/authorino/0.log"
Jul 24 14:25:34.450244 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:34.450149    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b8ksv_40c1d197-bad6-4963-a86a-30f31a209818/kuadrant-console-plugin/0.log"
Jul 24 14:25:34.787567 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:34.787532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nwdnl_75b2e2b1-7cce-4e32-99ab-a203716b8bab/manager/0.log"
Jul 24 14:25:36.972433 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:36.972382    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbfzr_9f938bc7-c7ce-4d4d-a383-2c2d7444fe09/node-exporter/0.log"
Jul 24 14:25:36.999217 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:36.999186    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbfzr_9f938bc7-c7ce-4d4d-a383-2c2d7444fe09/kube-rbac-proxy/0.log"
Jul 24 14:25:37.028171 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.028087    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbfzr_9f938bc7-c7ce-4d4d-a383-2c2d7444fe09/init-textfile/0.log"
Jul 24 14:25:37.074352 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.074315    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsmdb_18cbaf6f-ccc2-4330-b925-a09fffbbf140/kube-rbac-proxy-main/0.log"
Jul 24 14:25:37.105868 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.105832    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsmdb_18cbaf6f-ccc2-4330-b925-a09fffbbf140/kube-rbac-proxy-self/0.log"
Jul 24 14:25:37.139997 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.139550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rsmdb_18cbaf6f-ccc2-4330-b925-a09fffbbf140/openshift-state-metrics/0.log"
Jul 24 14:25:37.201688 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.201651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/prometheus/0.log"
Jul 24 14:25:37.254746 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.254717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/config-reloader/0.log"
Jul 24 14:25:37.292862 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.292773    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/thanos-sidecar/0.log"
Jul 24 14:25:37.341879 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.341847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/kube-rbac-proxy-web/0.log"
Jul 24 14:25:37.377067 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.377031    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/kube-rbac-proxy/0.log"
Jul 24 14:25:37.422197 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.422162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/kube-rbac-proxy-thanos/0.log"
Jul 24 14:25:37.451225 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.451195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_25edc33f-5339-4437-8ffb-cb6b02e8def7/init-config-reloader/0.log"
Jul 24 14:25:37.498741 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.498686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-htshn_df0dd117-9029-4910-9fe5-43ba7eae70f1/prometheus-operator/0.log"
Jul 24 14:25:37.525985 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.525953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-htshn_df0dd117-9029-4910-9fe5-43ba7eae70f1/kube-rbac-proxy/0.log"
Jul 24 14:25:37.721115 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.721083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/thanos-query/0.log"
Jul 24 14:25:37.751012 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.750981    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/kube-rbac-proxy-web/0.log"
Jul 24 14:25:37.789344 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.789311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/kube-rbac-proxy/0.log"
Jul 24 14:25:37.814227 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.814194    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/prom-label-proxy/0.log"
Jul 24 14:25:37.843572 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.843538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/kube-rbac-proxy-rules/0.log"
Jul 24 14:25:37.872159 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:37.872127    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f7bb66f4d-mh8rf_9719e502-889e-41ac-8721-fe00fcac7d7e/kube-rbac-proxy-metrics/0.log"
Jul 24 14:25:39.988041 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:39.988005    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"]
Jul 24 14:25:39.997131 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:39.997092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.007340 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.005890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"]
Jul 24 14:25:40.118180 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.118139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-proc\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.118456 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.118429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-sys\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.118671 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.118654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-podres\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.118863 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.118848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-lib-modules\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.119002 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.118988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jzt\" (UniqueName: \"kubernetes.io/projected/51001ba1-61f3-4ea8-977e-860146d2442a-kube-api-access-48jzt\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220200 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-lib-modules\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220200 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jzt\" (UniqueName: \"kubernetes.io/projected/51001ba1-61f3-4ea8-977e-860146d2442a-kube-api-access-48jzt\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-proc\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-sys\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-podres\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-lib-modules\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-proc\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-sys\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.220448 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.220429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51001ba1-61f3-4ea8-977e-860146d2442a-podres\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.232401 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.232369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jzt\" (UniqueName: \"kubernetes.io/projected/51001ba1-61f3-4ea8-977e-860146d2442a-kube-api-access-48jzt\") pod \"perf-node-gather-daemonset-ncgw6\" (UID: \"51001ba1-61f3-4ea8-977e-860146d2442a\") " pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.321541 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.321470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:40.511113 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.506986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"]
Jul 24 14:25:40.574365 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.574293    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57c658fdc4-944s4_1366fb54-56f9-41cc-b790-2410878f0bbc/console/0.log"
Jul 24 14:25:40.582003 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:40.581969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6" event={"ID":"51001ba1-61f3-4ea8-977e-860146d2442a","Type":"ContainerStarted","Data":"81791ed2d9b705b50c969ea60f9f88e58e00c9e2433fcc2b25d95fa6187493cd"}
Jul 24 14:25:41.589351 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:41.589302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6" event={"ID":"51001ba1-61f3-4ea8-977e-860146d2442a","Type":"ContainerStarted","Data":"e8d82fc8f5be33a27aaa55c10d4d3594692cbb48176926aefc03d9dfe82f121f"}
Jul 24 14:25:41.589753 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:41.589447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:41.610735 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:41.610684    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6" podStartSLOduration=2.610667052 podStartE2EDuration="2.610667052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:41.609000068 +0000 UTC m=+3081.071606913" watchObservedRunningTime="2026-07-24 14:25:41.610667052 +0000 UTC m=+3081.073273896"
Jul 24 14:25:42.377999 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:42.377970    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sknvp_1bdf7870-c76d-4f29-9b35-fe6f925e1ba1/dns/0.log"
Jul 24 14:25:42.401033 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:42.401008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sknvp_1bdf7870-c76d-4f29-9b35-fe6f925e1ba1/kube-rbac-proxy/0.log"
Jul 24 14:25:42.453442 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:42.453408    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dwt4j_fd7ee460-ed31-4b90-97f6-39b857c5dcf2/dns-node-resolver/0.log"
Jul 24 14:25:43.064021 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:43.063988    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7qndb_21c3ce53-0fde-450c-b0f0-28920045b672/node-ca/0.log"
Jul 24 14:25:44.338902 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:44.338875    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85dc78bf9-jflxb_feb7105e-55dc-4bf2-a756-22626d6c0e72/kube-auth-proxy/0.log"
Jul 24 14:25:45.370644 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:45.370609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9dg7q_7584b3a9-a5ee-4a4f-8e60-41e3231f9e2c/serve-healthcheck-canary/0.log"
Jul 24 14:25:45.976490 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:45.976459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhpbl_d4b30834-dcf5-416b-b746-ca66c3463d32/kube-rbac-proxy/0.log"
Jul 24 14:25:46.028861 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:46.028835    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhpbl_d4b30834-dcf5-416b-b746-ca66c3463d32/exporter/0.log"
Jul 24 14:25:46.071333 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:46.071308    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhpbl_d4b30834-dcf5-416b-b746-ca66c3463d32/extractor/0.log"
Jul 24 14:25:47.605970 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:47.605940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hh2p5/perf-node-gather-daemonset-ncgw6"
Jul 24 14:25:49.627210 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:49.627179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-bvcb6_5a067aa3-05ca-4af7-88a5-89e5f04ad91e/manager/0.log"
Jul 24 14:25:49.659042 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:49.659015    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85fcbb566-bvcb6_5a067aa3-05ca-4af7-88a5-89e5f04ad91e/copy-manifests/0.log"
Jul 24 14:25:50.647911 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:50.647874    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85dc78bf9-jflxb_feb7105e-55dc-4bf2-a756-22626d6c0e72/kube-auth-proxy/0.log"
Jul 24 14:25:50.988226 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:50.988152    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d64c44479-p6hq5_884c3949-0d63-4988-9aa4-6a923533c921/manager/0.log"
Jul 24 14:25:51.049796 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:51.049764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-glv9t_96fe3022-0140-4921-8d4d-5afaad8ac8be/openshift-lws-operator/0.log"
Jul 24 14:25:52.341229 ip-10-0-136-88 kubenswrapper[2579]: I0724 14:25:52.341196    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85dc78bf9-jflxb_feb7105e-55dc-4bf2-a756-22626d6c0e72/kube-auth-proxy/0.log"