Jul 10 17:38:52.828314 ip-10-0-133-236 systemd[1]: Starting Kubernetes Kubelet... Jul 10 17:38:53.216727 ip-10-0-133-236 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:38:53.216727 ip-10-0-133-236 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 17:38:53.216727 ip-10-0-133-236 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:38:53.216727 ip-10-0-133-236 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 17:38:53.216727 ip-10-0-133-236 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 17:38:53.218835 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.218748 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 17:38:53.222322 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222301 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.222322 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222319 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.222322 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222324 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.222322 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222328 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222332 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222336 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222339 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222343 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222347 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222351 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222355 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222358 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222362 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222371 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222375 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222378 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222382 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222386 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222390 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222394 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222398 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222401 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.222567 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222405 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222409 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222413 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222417 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222421 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222427 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222432 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222437 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222441 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222445 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222450 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222454 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222459 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222463 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222467 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222473 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222478 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222482 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222486 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.223269 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222490 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222494 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222498 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222503 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222507 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222512 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222516 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222521 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222525 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222529 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222533 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222536 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222541 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222545 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222549 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222554 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222558 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222564 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222570 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222575 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.223757 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222580 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222584 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222589 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222593 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222598 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222602 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222606 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222611 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222617 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222625 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222630 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222634 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222639 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222644 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222649 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222653 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222657 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222661 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222666 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.224316 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222671 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222675 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222679 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222683 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222687 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222692 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222697 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222701 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222705 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222709 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222713 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222717 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222722 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222726 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222730 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.222734 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.223091 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.223114 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.224887 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.223121 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.225980 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.225964 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.225980 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.225979 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.225983 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.225986 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226008 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226011 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226014 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226016 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226019 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226021 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226024 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226027 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226029 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226031 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226035 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226037 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226040 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226043 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226045 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226048 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226050 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.226071 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226053 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226056 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226058 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226060 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226063 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226065 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226068 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226070 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226073 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226076 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226079 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226082 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226085 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226087 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226090 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226092 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226095 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226097 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226101 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.226565 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226104 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226107 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226110 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226113 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226116 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226118 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226121 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226124 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226128 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226131 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226133 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226136 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226138 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226141 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226143 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226147 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226150 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226153 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226155 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226158 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.227079 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226161 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226163 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226166 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226168 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226171 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226173 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226175 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226178 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226180 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226183 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226185 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226187 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226190 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226192 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226194 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226197 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226200 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226202 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226204 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226208 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.227564 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226212 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226216 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226219 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226222 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226225 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226227 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226230 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226234 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226236 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226239 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226242 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226245 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226248 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226251 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226254 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226256 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226259 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226262 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226264 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226639 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.228081 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226644 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226646 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226649 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226652 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226654 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226656 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226659 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226662 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226664 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226667 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226670 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226672 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226675 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226678 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226680 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226683 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226685 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226688 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226690 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226694 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.228578 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226697 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226699 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226702 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226705 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226707 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226709 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226712 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226714 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226717 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226719 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226721 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226724 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226726 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226730 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226733 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226736 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226738 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226740 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226743 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.229085 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226745 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226747 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226750 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226752 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226755 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226757 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226761 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226763 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226765 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226769 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226772 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226775 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226779 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226782 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226784 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226787 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226789 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226792 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.229592 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226796 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226799 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226802 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226805 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226807 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226810 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226813 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226816 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226819 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226821 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226824 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226826 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226828 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226831 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226833 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226835 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226838 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226840 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226843 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226845 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.230044 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226847 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226850 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226853 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226855 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226858 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226860 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226864 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226866 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226869 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226872 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226874 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226877 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226880 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226882 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226885 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226887 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226889 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226892 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226894 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226896 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.230525 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.226899 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.226973 2582 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.226980 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.226986 2582 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227005 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227009 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227012 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227024 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227028 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227031 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227034 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227037 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227040 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227043 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227047 2582 flags.go:64] FLAG: --cgroup-root="" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227049 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227053 2582 flags.go:64] FLAG: --client-ca-file="" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227055 2582 flags.go:64] FLAG: --cloud-provider="external" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227058 2582 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227063 2582 flags.go:64] FLAG: --cluster-domain="" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227066 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227069 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227071 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227075 2582 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 17:38:53.231047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227079 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227082 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227085 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227088 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227090 2582 flags.go:64] FLAG: --contention-profiling="false" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227094 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227096 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227100 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227102 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227106 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227109 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227111 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227114 2582 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227116 2582 flags.go:64] FLAG: --enable-server="true" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227119 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227123 2582 flags.go:64] FLAG: --event-burst="100" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227126 2582 flags.go:64] FLAG: --event-qps="50" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227129 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227132 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227135 2582 flags.go:64] FLAG: --eviction-hard="" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227139 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227142 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227144 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227148 2582 flags.go:64] FLAG: --eviction-soft="" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227151 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 17:38:53.231622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227154 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227159 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227162 2582 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227164 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227167 2582 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227170 2582 flags.go:64] FLAG: --feature-gates="" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227173 2582 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227176 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227179 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227181 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227185 2582 flags.go:64] FLAG: --healthz-port="10248" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227188 2582 flags.go:64] FLAG: --help="false" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227191 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227194 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227196 2582 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227199 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227203 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227206 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227208 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227211 2582 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227214 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227217 2582 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227219 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227222 2582 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 17:38:53.232345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227225 2582 flags.go:64] FLAG: --kube-reserved="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227228 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227231 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227234 2582 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227236 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227239 2582 flags.go:64] FLAG: --lock-file="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227241 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227245 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227247 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227254 2582 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227257 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227259 2582 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227262 2582 flags.go:64] FLAG: --logging-format="text" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227265 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227268 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227271 2582 flags.go:64] FLAG: --manifest-url="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227273 2582 flags.go:64] FLAG: --manifest-url-header="" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227277 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227281 2582 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227289 2582 flags.go:64] FLAG: --max-pods="110" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227292 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227295 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227297 2582 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227300 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227303 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 17:38:53.232956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227306 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227309 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227315 2582 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227318 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227322 2582 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227324 2582 flags.go:64] FLAG: --pod-cidr="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227327 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227332 2582 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227335 2582 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227338 2582 flags.go:64] FLAG: --pods-per-core="0" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227341 2582 flags.go:64] FLAG: --port="10250" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227344 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227346 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0158f9462ac45ee3a" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227350 2582 flags.go:64] FLAG: --qos-reserved="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227352 2582 flags.go:64] FLAG: --read-only-port="10255" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227356 2582 flags.go:64] FLAG: --register-node="true" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227359 2582 flags.go:64] FLAG: --register-with-taints="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227364 2582 flags.go:64] FLAG: --registry-burst="10" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227367 2582 flags.go:64] FLAG: --registry-qps="5" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227369 2582 flags.go:64] FLAG: --reserved-cpus="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227372 2582 flags.go:64] FLAG: --reserved-memory="" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227376 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227378 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227381 2582 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227384 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 17:38:53.233626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227387 2582 flags.go:64] FLAG: --runonce="false" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227389 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227392 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227395 2582 flags.go:64] FLAG: --seccomp-default="false" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227398 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227400 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227403 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227406 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227409 2582 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227411 2582 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227414 2582 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227417 2582 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227419 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227422 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227425 2582 flags.go:64] FLAG: --system-cgroups="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227428 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227433 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227435 2582 flags.go:64] FLAG: --tls-cert-file="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227438 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227442 2582 flags.go:64] FLAG: --tls-min-version="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227444 2582 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227447 2582 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227449 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227453 2582 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227457 2582 flags.go:64] FLAG: --v="2" Jul 10 17:38:53.234256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227460 2582 flags.go:64] FLAG: --version="false" Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227464 2582 flags.go:64] FLAG: --vmodule="" Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227467 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227470 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227608 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227613 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227616 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227619 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227621 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227624 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227626 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227629 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227632 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227634 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227638 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227642 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227645 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227648 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227651 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227653 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.234863 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227655 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227658 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227660 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227663 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227665 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227667 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227681 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227685 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227687 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227690 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227692 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227696 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227699 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227701 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227704 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227706 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227709 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227711 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227714 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227717 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.235371 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227719 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227722 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227724 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227726 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227730 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227733 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227737 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227739 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227742 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227745 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227747 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227750 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227752 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227754 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227757 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227759 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227761 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227764 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227766 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227769 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.235856 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227771 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227774 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227776 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227780 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227783 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227786 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227788 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227791 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227793 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227796 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227799 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227801 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227804 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227807 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227809 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227811 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227814 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227816 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227818 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227821 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.236366 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227823 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227826 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227828 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227830 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227833 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227835 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227837 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227840 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227842 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227844 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227847 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227849 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227852 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227856 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227859 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227863 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227865 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227868 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227871 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227874 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.236861 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227877 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227881 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.227883 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.227891 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.234530 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.234547 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.235454 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.235463 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235519 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235526 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235530 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235533 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235536 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235539 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.237363 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235542 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235545 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235548 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235551 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235554 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235556 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235559 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235562 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235564 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235567 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235570 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235572 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235575 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235577 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235580 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235582 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235585 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235589 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235592 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.237717 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235595 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235598 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235601 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235604 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235609 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235611 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235614 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235617 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235620 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235623 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235626 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235629 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235632 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235634 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235637 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235639 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235642 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235644 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235647 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235649 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.238196 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235652 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235654 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235657 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235659 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235661 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235664 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235666 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235668 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235671 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235673 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235675 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235678 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235681 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235684 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235688 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235691 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235694 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235697 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235700 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.238703 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235703 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235706 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235708 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235711 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235713 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235716 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235718 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235721 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235723 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235725 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235728 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235730 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235732 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235735 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235737 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235740 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235742 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235745 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235747 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235749 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.239178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235751 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235754 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235756 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235758 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235761 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235764 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235766 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235769 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235771 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235774 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235777 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235779 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235782 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235784 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235787 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.239656 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.235792 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235901 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235905 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235908 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235911 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235914 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235916 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235919 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235921 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235924 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235927 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235929 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235932 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235934 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235937 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235940 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235942 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235944 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235947 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235949 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 17:38:53.240052 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235951 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235954 2582 feature_gate.go:328] unrecognized feature gate: Example Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235956 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235960 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235963 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235966 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235968 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235971 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235973 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235976 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235979 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235981 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235983 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.235986 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236004 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236008 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236010 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236013 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236015 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 17:38:53.240539 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236018 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236021 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236023 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236025 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236028 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236030 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236033 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236035 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236037 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236040 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236042 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236045 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236047 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236050 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236052 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236055 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236057 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236060 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236062 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236065 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 17:38:53.241013 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236067 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236070 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236072 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236076 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236078 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236080 2582 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236083 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236085 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236088 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236090 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236092 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236095 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236097 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236099 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236102 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236104 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236107 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236109 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236113 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236116 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 17:38:53.241488 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236119 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236122 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236125 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236128 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236131 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236134 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236137 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236140 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236142 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236145 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236147 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236150 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236152 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236155 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236157 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236160 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236163 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236166 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236168 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 17:38:53.241976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236171 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:53.236173 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.236178 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.236283 2582 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.238301 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.239708 2582 server.go:1019] "Starting client certificate rotation" Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.239808 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.239837 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 17:38:53.242489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.242175 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:38:53.244789 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.244577 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 17:38:53.253830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.253814 2582 log.go:25] "Validated CRI v1 runtime API" Jul 10 17:38:53.257018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.257004 2582 log.go:25] "Validated CRI v1 image API" Jul 10 17:38:53.258801 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.258788 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 17:38:53.261105 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.261089 2582 fs.go:135] Filesystem UUIDs: map[17556c12-9697-4a40-8553-c086386297a4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb1baa16-1324-4c84-b7fd-9baeb0ae74a1:/dev/nvme0n1p3] Jul 10 17:38:53.261148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.261105 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 17:38:53.266422 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.266322 2582 manager.go:216] Machine: {Timestamp:2026-07-10 17:38:53.265163607 +0000 UTC m=+0.325978933 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101276 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a096f79fadf2103c259541ca266a SystemUUID:ec27a096-f79f-adf2-103c-259541ca266a BootID:c28eb3d7-b0d9-4885-a5d9-087b09d004ab Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:01:45:54:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:01:45:54:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:e8:02:c0:a5:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 17:38:53.266422 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.266421 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 17:38:53.266527 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.266492 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 17:38:53.267998 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.267962 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 17:38:53.268138 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.268004 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 17:38:53.268652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.268642 2582 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 17:38:53.268688 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.268653 2582 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 17:38:53.268688 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.268661 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:38:53.269373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.269362 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 17:38:53.270603 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.270593 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 17:38:53.270798 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.270783 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:38:53.273225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273213 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:38:53.273330 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273321 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 17:38:53.273361 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273349 2582 kubelet.go:486] "Attempting to sync node with API server" Jul 10 17:38:53.273361 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273358 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 17:38:53.273414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273377 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 17:38:53.273414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273386 2582 kubelet.go:398] "Adding apiserver pod source" Jul 10 17:38:53.273414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273394 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 17:38:53.273839 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273822 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:38:53.273943 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.273848 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 17:38:53.276203 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.276175 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 17:38:53.278106 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278087 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 17:38:53.278169 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278121 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 17:38:53.278318 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278306 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 17:38:53.278364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278324 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 17:38:53.278364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278329 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 17:38:53.278364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278335 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 17:38:53.278364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278343 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 17:38:53.278364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278348 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 17:38:53.278822 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278814 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 17:38:53.278822 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278822 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 17:38:53.278879 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278832 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 17:38:53.278879 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278837 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 17:38:53.278879 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278845 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 17:38:53.278879 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.278851 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 17:38:53.279552 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.279542 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 17:38:53.279552 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.279552 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 17:38:53.282557 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.282542 2582 server.go:1268] "Started kubelet" Jul 10 17:38:53.282661 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.282634 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 17:38:53.286568 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.286319 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 17:38:53.286815 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.286801 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 17:38:53.286820 ip-10-0-133-236 systemd[1]: Started Kubernetes Kubelet. Jul 10 17:38:53.287947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.287907 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 17:38:53.289143 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.289116 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 17:38:53.289410 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.289377 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 17:38:53.289507 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.289419 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 17:38:53.290060 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.290039 2582 server.go:310] "Adding debug handlers to kubelet server" Jul 10 17:38:53.292412 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.292386 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 17:38:53.294225 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.293334 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-236.ec2.internal.18c0fe5e45585ad9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-236.ec2.internal,UID:ip-10-0-133-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-236.ec2.internal,},FirstTimestamp:2026-07-10 17:38:53.282515673 +0000 UTC m=+0.343330998,LastTimestamp:2026-07-10 17:38:53.282515673 +0000 UTC m=+0.343330998,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-236.ec2.internal,}" Jul 10 17:38:53.296557 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.296539 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 17:38:53.296639 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.296573 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 17:38:53.297140 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.297024 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.297347 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297334 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 17:38:53.297455 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297335 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 17:38:53.297455 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297455 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 17:38:53.297594 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297355 2582 factory.go:55] Registering systemd factory Jul 10 17:38:53.297594 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297526 2582 factory.go:223] Registration of the systemd container factory successfully Jul 10 17:38:53.297594 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297561 2582 reconstruct.go:97] "Volume reconstruction finished" Jul 10 17:38:53.297594 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297571 2582 reconciler.go:29] "Reconciler: start to sync state" Jul 10 17:38:53.297756 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297732 2582 factory.go:153] Registering CRI-O factory Jul 10 17:38:53.297756 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297744 2582 factory.go:223] Registration of the crio container factory successfully Jul 10 17:38:53.297844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297791 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 17:38:53.297844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297812 2582 factory.go:103] Registering Raw factory Jul 10 17:38:53.297844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.297826 2582 manager.go:1196] Started watching for new ooms in manager Jul 10 17:38:53.298354 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.298340 2582 manager.go:349] Starting recovery of all containers Jul 10 17:38:53.298621 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.298596 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 17:38:53.298794 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.298769 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 17:38:53.299450 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.299424 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kqs5" Jul 10 17:38:53.306544 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.306523 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kqs5" Jul 10 17:38:53.310200 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.310182 2582 manager.go:354] Recovery completed Jul 10 17:38:53.312055 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.312037 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 10 17:38:53.314755 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.314743 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.316449 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.316433 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.316513 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.316464 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.316513 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.316477 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.317040 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317025 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 17:38:53.317040 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317036 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.317040 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317039 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 17:38:53.317172 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317059 2582 state_mem.go:36] "Initialized new in-memory state store" Jul 10 17:38:53.317706 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317691 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.317812 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317716 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.317965 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.317949 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.318235 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.318222 2582 policy_none.go:49] "None policy: Start" Jul 10 17:38:53.318377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.318354 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 17:38:53.318471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.318385 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 17:38:53.319585 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.319570 2582 policy_none.go:47] "Start" Jul 10 17:38:53.359945 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.359930 2582 manager.go:337] "Starting Device Plugin manager" Jul 10 17:38:53.360028 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.359965 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 17:38:53.360028 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.359979 2582 server.go:87] "Starting device plugin registration server" Jul 10 17:38:53.360279 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.360268 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 17:38:53.360319 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.360284 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 17:38:53.360394 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.360375 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 17:38:53.360522 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.360482 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 17:38:53.360522 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.360522 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 17:38:53.360939 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.360874 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 17:38:53.360939 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.360920 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.425014 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.424978 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 17:38:53.426298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.426285 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 17:38:53.426368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.426321 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 17:38:53.426368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.426342 2582 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 17:38:53.426633 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.426371 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 17:38:53.428692 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.428676 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:38:53.460572 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.460551 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.461783 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.461768 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.461860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.461802 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.461860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.461816 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.461860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.461843 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.468091 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.468050 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.468091 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.468071 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-236.ec2.internal\": node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.479241 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.479225 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.527154 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.527130 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal"] Jul 10 17:38:53.527232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.527186 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.528497 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.528479 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.528567 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.528506 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.528567 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.528521 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.529761 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.529748 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.529913 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.529900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.529951 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.529929 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.530456 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530437 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.530516 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530465 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.530516 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530475 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.530590 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530438 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.530590 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530536 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.530590 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.530546 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.531674 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.531661 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.531714 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.531685 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 17:38:53.532405 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.532371 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 10 17:38:53.532405 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.532405 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 17:38:53.532513 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.532418 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 10 17:38:53.555560 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.555536 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.ec2.internal\" not found" node="ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.559915 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.559899 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.ec2.internal\" not found" node="ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.579724 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.579703 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.599399 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.599371 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a736e251cd3abf42f7ab6349dd18630-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"3a736e251cd3abf42f7ab6349dd18630\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.599399 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.599399 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.599525 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.599424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.680465 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.680441 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.699782 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.699868 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.699868 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a736e251cd3abf42f7ab6349dd18630-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"3a736e251cd3abf42f7ab6349dd18630\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.699950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.699950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a736e251cd3abf42f7ab6349dd18630-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"3a736e251cd3abf42f7ab6349dd18630\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.699950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.699866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7494cf2ba7496408a828f6dd070ca22-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"c7494cf2ba7496408a828f6dd070ca22\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.781248 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.781178 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.857763 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.857735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.862772 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:53.862755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:53.881459 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.881415 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:53.981929 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:53.981899 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.082476 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.082409 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.182876 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.182853 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.239325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.239303 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 17:38:54.239792 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.239424 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 17:38:54.283671 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.283648 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.287072 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.287054 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:38:54.309797 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.309766 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 17:33:53 +0000 UTC" deadline="2028-02-07 22:00:42.400676482 +0000 UTC" Jul 10 17:38:54.309797 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.309795 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13852h21m48.090884738s" Jul 10 17:38:54.360611 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:54.360581 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a736e251cd3abf42f7ab6349dd18630.slice/crio-db266d6a59a390a39d007a530ffcdf4eb57ed889c09ca13f01059d55a66add68 WatchSource:0}: Error finding container db266d6a59a390a39d007a530ffcdf4eb57ed889c09ca13f01059d55a66add68: Status 404 returned error can't find the container with id db266d6a59a390a39d007a530ffcdf4eb57ed889c09ca13f01059d55a66add68 Jul 10 17:38:54.361099 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:54.361078 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7494cf2ba7496408a828f6dd070ca22.slice/crio-f81f6a843a16b1515cfd7db592d7dea26c61252223f6bdba982ffd35c167c51d WatchSource:0}: Error finding container f81f6a843a16b1515cfd7db592d7dea26c61252223f6bdba982ffd35c167c51d: Status 404 returned error can't find the container with id f81f6a843a16b1515cfd7db592d7dea26c61252223f6bdba982ffd35c167c51d Jul 10 17:38:54.365908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.365894 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:38:54.384548 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.384527 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.429954 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.429910 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"c7494cf2ba7496408a828f6dd070ca22","Type":"ContainerStarted","Data":"f81f6a843a16b1515cfd7db592d7dea26c61252223f6bdba982ffd35c167c51d"} Jul 10 17:38:54.430896 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.430876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" event={"ID":"3a736e251cd3abf42f7ab6349dd18630","Type":"ContainerStarted","Data":"db266d6a59a390a39d007a530ffcdf4eb57ed889c09ca13f01059d55a66add68"} Jul 10 17:38:54.485043 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.485022 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.585580 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.585556 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.686491 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:54.686463 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 10 17:38:54.770099 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.770067 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:38:54.797093 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.797067 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 10 17:38:54.808049 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.808030 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:38:54.810439 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.810425 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 10 17:38:54.818750 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.818732 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 17:38:54.828450 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:54.828423 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:38:55.274399 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.274365 2582 apiserver.go:52] "Watching apiserver" Jul 10 17:38:55.282199 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.282165 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 17:38:55.284432 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.284398 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-72p97","openshift-network-diagnostics/network-check-target-gt5xj","kube-system/global-pull-secret-syncer-gwzp8","openshift-image-registry/node-ca-6xr2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal","openshift-multus/multus-fr257","openshift-multus/network-metrics-daemon-wxtvv","openshift-network-operator/iptables-alerter-qq7vv","openshift-ovn-kubernetes/ovnkube-node-phwsw","kube-system/konnectivity-agent-lgddx","kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm","openshift-cluster-node-tuning-operator/tuned-zpstz","openshift-dns/node-resolver-4b7hj"] Jul 10 17:38:55.286234 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.286210 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.286339 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.286297 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:38:55.288274 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.287566 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.289331 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.289303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.289431 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.289393 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.291276 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.291604 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.291695 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gdrv8\"" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.291743 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.292440 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.292498 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.293524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.292593 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.294660 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.294341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fr257" Jul 10 17:38:55.295025 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.294963 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.296489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.296470 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.296843 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.296820 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 17:38:55.297319 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.297297 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.298949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.298152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:55.298949 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.298213 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:38:55.298949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.298312 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 17:38:55.298949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.298488 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 17:38:55.298949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.298740 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dp9rw\"" Jul 10 17:38:55.299570 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299437 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 17:38:55.299709 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nswpd\"" Jul 10 17:38:55.299782 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299712 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.299860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299843 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 17:38:55.299913 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299858 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 17:38:55.299993 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.299970 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 17:38:55.300164 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.300138 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nkqhq\"" Jul 10 17:38:55.300234 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.300215 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 17:38:55.300372 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.300345 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rf7x\"" Jul 10 17:38:55.300833 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.300809 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.301276 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.301173 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.301415 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.301396 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 17:38:55.301664 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.301645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z8qz6\"" Jul 10 17:38:55.301926 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.301908 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.302199 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.303131 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.303483 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.303934 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.304676 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.305600 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.305423 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.307144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.306552 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.307533 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.307490 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 17:38:55.307789 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.307769 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6l57c\"" Jul 10 17:38:55.307870 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.307836 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 17:38:55.308163 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-kubelet\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.308275 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308168 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-slash\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.308275 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-log-socket\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.308275 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308222 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.308275 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308239 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.308491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308310 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-conf\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.308491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.308491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-multus-certs\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.308491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-systemd-units\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.308491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.308758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mnz\" (UniqueName: \"kubernetes.io/projected/7574bc68-924a-4b78-8734-d00609e93f71-kube-api-access-p7mnz\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.308758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308601 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnjf7\" (UniqueName: \"kubernetes.io/projected/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-kube-api-access-mnjf7\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.308758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308669 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-env-overrides\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.308758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-os-release\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-etc-kubernetes\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-netns\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308843 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-cni-binary-copy\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-tmp\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.308968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsp5\" (UniqueName: \"kubernetes.io/projected/49c2a170-d162-4a91-959f-5dd6e133855c-kube-api-access-pxsp5\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.309385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.308990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cnibin\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.309385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.309055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-system-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.309385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.309138 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-daemon-config\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.309385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.309367 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 17:38:55.309751 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.309700 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-glwbs\"" Jul 10 17:38:55.310446 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-systemd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.310533 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.310533 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-config\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.310533 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovn-node-metrics-cert\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-kubernetes\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-kubelet-config\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310598 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-systemd\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310609 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-hostroot\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-cnibin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310668 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-bin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-netd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-script-lib\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310739 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-serviceca\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310748 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9rnfm\"" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zspsk\" (UniqueName: \"kubernetes.io/projected/480f4dd5-3168-4d70-bfd4-630cc225498a-kube-api-access-zspsk\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310811 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-ovn\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-lib-modules\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49c2a170-d162-4a91-959f-5dd6e133855c-host-slash\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.311548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-os-release\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-node-log\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-var-lib-kubelet\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.310982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-system-cni-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311101 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-etc-tuned\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-host\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311179 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddf7\" (UniqueName: \"kubernetes.io/projected/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-kube-api-access-fddf7\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-socket-dir-parent\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311317 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-netns\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311346 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-etc-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp8ct\" (UniqueName: \"kubernetes.io/projected/62f47418-60ee-40ac-83f7-9b62d2a18531-kube-api-access-wp8ct\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.312502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-dbus\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49c2a170-d162-4a91-959f-5dd6e133855c-iptables-alerter-script\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-kubelet\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311546 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpphb\" (UniqueName: \"kubernetes.io/projected/a092006b-cbfc-4c4a-81f0-a10014c00c2d-kube-api-access-fpphb\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-run\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-var-lib-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311650 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311676 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysconfig\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-multus\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-conf-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311753 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-bin\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-modprobe-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-host\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-k8s-cni-cncf-io\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-sys\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.311923 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:55.313328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.312823 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 17:38:55.331141 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.331107 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdntt" Jul 10 17:38:55.338508 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.338487 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdntt" Jul 10 17:38:55.398392 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.398365 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-multus-certs\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-systemd-units\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412316 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mnz\" (UniqueName: \"kubernetes.io/projected/7574bc68-924a-4b78-8734-d00609e93f71-kube-api-access-p7mnz\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412431 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjf7\" (UniqueName: \"kubernetes.io/projected/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-kube-api-access-mnjf7\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-multus-certs\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-env-overrides\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-os-release\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-etc-kubernetes\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-netns\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-cni-binary-copy\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-tmp\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412656 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.414493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsp5\" (UniqueName: \"kubernetes.io/projected/49c2a170-d162-4a91-959f-5dd6e133855c-kube-api-access-pxsp5\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cnibin\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-system-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-daemon-config\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-systemd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-config\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.412964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovn-node-metrics-cert\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-kubernetes\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-kubelet-config\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-system-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cnibin\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-env-overrides\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-kubelet-config\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-systemd-units\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-os-release\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413405 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-etc-kubernetes\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.415479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413445 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-netns\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.413968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-daemon-config\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-systemd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414095 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414133 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414256 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-systemd\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlpg\" (UniqueName: \"kubernetes.io/projected/f55360d7-4733-4c29-9eed-31ebb15baeaa-kube-api-access-jxlpg\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-agent-certs\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35f62177-db28-40d7-a0c6-a2aafe07c415-hosts-file\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414565 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kht\" (UniqueName: \"kubernetes.io/projected/35f62177-db28-40d7-a0c6-a2aafe07c415-kube-api-access-d2kht\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-hostroot\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-cnibin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-bin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-netd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-script-lib\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-config\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.416363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-serviceca\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414836 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-kubernetes\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414911 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zspsk\" (UniqueName: \"kubernetes.io/projected/480f4dd5-3168-4d70-bfd4-630cc225498a-kube-api-access-zspsk\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.414952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-ovn\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415035 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-systemd\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415082 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-bin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-ovn\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415155 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-lib-modules\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415185 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49c2a170-d162-4a91-959f-5dd6e133855c-host-slash\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-registration-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415243 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-os-release\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-node-log\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-var-lib-kubelet\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-system-cni-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415357 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.417297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415377 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-etc-tuned\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-host\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddf7\" (UniqueName: \"kubernetes.io/projected/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-kube-api-access-fddf7\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/480f4dd5-3168-4d70-bfd4-630cc225498a-cni-binary-copy\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-socket-dir-parent\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-netns\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-etc-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-socket-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415697 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415756 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp8ct\" (UniqueName: \"kubernetes.io/projected/62f47418-60ee-40ac-83f7-9b62d2a18531-kube-api-access-wp8ct\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415784 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-dbus\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovnkube-script-lib\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49c2a170-d162-4a91-959f-5dd6e133855c-iptables-alerter-script\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415838 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-kubelet\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpphb\" (UniqueName: \"kubernetes.io/projected/a092006b-cbfc-4c4a-81f0-a10014c00c2d-kube-api-access-fpphb\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-run\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-sys-fs\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-var-lib-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.415991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysconfig\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416090 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-multus\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416099 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-socket-dir-parent\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-conf-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-bin\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-cnibin\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-modprobe-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-host\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f62177-db28-40d7-a0c6-a2aafe07c415-tmp-dir\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-k8s-cni-cncf-io\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-sys\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:55.418899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416366 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-device-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-kubelet\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416424 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-slash\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416475 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-slash\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-serviceca\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-log-socket\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-log-socket\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-host\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416593 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-netd\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416628 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-netns\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.416681 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/49c2a170-d162-4a91-959f-5dd6e133855c-host-slash\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416749 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-lib-modules\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416819 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-node-log\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-cni-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.416867 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:38:55.916846099 +0000 UTC m=+2.977661440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.419748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416939 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-os-release\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-var-lib-kubelet\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.416980 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysconfig\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-conf\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/49c2a170-d162-4a91-959f-5dd6e133855c-iptables-alerter-script\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417097 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-konnectivity-ca\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-sys\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417294 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-sysctl-conf\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417399 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/542ae712-d1af-4d6d-99ee-60aeb92da71f-dbus\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417474 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-kubelet\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-cni-bin\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-system-cni-dir\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-multus-conf-dir\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-run-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417837 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-host-kubelet\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417843 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-run-k8s-cni-cncf-io\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.417850 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-etc-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.420707 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.417948 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-hostroot\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418083 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-host\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/480f4dd5-3168-4d70-bfd4-630cc225498a-host-var-lib-cni-multus\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418467 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-run\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.418511 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:55.918488473 +0000 UTC m=+2.979303788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418538 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a092006b-cbfc-4c4a-81f0-a10014c00c2d-var-lib-openvswitch\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418565 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7574bc68-924a-4b78-8734-d00609e93f71-etc-modprobe-d\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.421633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.418841 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-tmp\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.425225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.425142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7574bc68-924a-4b78-8734-d00609e93f71-etc-tuned\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.425417 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a092006b-cbfc-4c4a-81f0-a10014c00c2d-ovn-node-metrics-cert\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.425434 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjf7\" (UniqueName: \"kubernetes.io/projected/2c38fcda-c9c4-4bd0-85e6-4ce5294b3150-kube-api-access-mnjf7\") pod \"node-ca-6xr2w\" (UID: \"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150\") " pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.425677 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.425707 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.425722 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:55.430510 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.425968 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:55.925927155 +0000 UTC m=+2.986742485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:55.436196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.436172 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsp5\" (UniqueName: \"kubernetes.io/projected/49c2a170-d162-4a91-959f-5dd6e133855c-kube-api-access-pxsp5\") pod \"iptables-alerter-qq7vv\" (UID: \"49c2a170-d162-4a91-959f-5dd6e133855c\") " pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.437205 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.437178 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mnz\" (UniqueName: \"kubernetes.io/projected/7574bc68-924a-4b78-8734-d00609e93f71-kube-api-access-p7mnz\") pod \"tuned-zpstz\" (UID: \"7574bc68-924a-4b78-8734-d00609e93f71\") " pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.437492 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.437439 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddf7\" (UniqueName: \"kubernetes.io/projected/d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2-kube-api-access-fddf7\") pod \"multus-additional-cni-plugins-72p97\" (UID: \"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2\") " pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.437611 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.437594 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpphb\" (UniqueName: \"kubernetes.io/projected/a092006b-cbfc-4c4a-81f0-a10014c00c2d-kube-api-access-fpphb\") pod \"ovnkube-node-phwsw\" (UID: \"a092006b-cbfc-4c4a-81f0-a10014c00c2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.438327 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.438292 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp8ct\" (UniqueName: \"kubernetes.io/projected/62f47418-60ee-40ac-83f7-9b62d2a18531-kube-api-access-wp8ct\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.438641 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.438618 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zspsk\" (UniqueName: \"kubernetes.io/projected/480f4dd5-3168-4d70-bfd4-630cc225498a-kube-api-access-zspsk\") pod \"multus-fr257\" (UID: \"480f4dd5-3168-4d70-bfd4-630cc225498a\") " pod="openshift-multus/multus-fr257" Jul 10 17:38:55.517844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-device-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.517844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-konnectivity-ca\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517882 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlpg\" (UniqueName: \"kubernetes.io/projected/f55360d7-4733-4c29-9eed-31ebb15baeaa-kube-api-access-jxlpg\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-device-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-agent-certs\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517973 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35f62177-db28-40d7-a0c6-a2aafe07c415-hosts-file\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kht\" (UniqueName: \"kubernetes.io/projected/35f62177-db28-40d7-a0c6-a2aafe07c415-kube-api-access-d2kht\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.518077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.517996 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518544 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35f62177-db28-40d7-a0c6-a2aafe07c415-hosts-file\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-registration-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-registration-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-socket-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-sys-fs\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-sys-fs\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f62177-db28-40d7-a0c6-a2aafe07c415-tmp-dir\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-socket-dir\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f55360d7-4733-4c29-9eed-31ebb15baeaa-etc-selinux\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.518861 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-konnectivity-ca\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.519466 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.518997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f62177-db28-40d7-a0c6-a2aafe07c415-tmp-dir\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.520744 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.520722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1bfe168c-9893-4b2e-a197-6943e3f9ec2d-agent-certs\") pod \"konnectivity-agent-lgddx\" (UID: \"1bfe168c-9893-4b2e-a197-6943e3f9ec2d\") " pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.526359 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.526294 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlpg\" (UniqueName: \"kubernetes.io/projected/f55360d7-4733-4c29-9eed-31ebb15baeaa-kube-api-access-jxlpg\") pod \"aws-ebs-csi-driver-node-msbvm\" (UID: \"f55360d7-4733-4c29-9eed-31ebb15baeaa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.526786 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.526764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kht\" (UniqueName: \"kubernetes.io/projected/35f62177-db28-40d7-a0c6-a2aafe07c415-kube-api-access-d2kht\") pod \"node-resolver-4b7hj\" (UID: \"35f62177-db28-40d7-a0c6-a2aafe07c415\") " pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.605141 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.605106 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qq7vv" Jul 10 17:38:55.615139 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.615110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-72p97" Jul 10 17:38:55.625430 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.625406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6xr2w" Jul 10 17:38:55.634788 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.634769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fr257" Jul 10 17:38:55.641730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.641693 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:38:55.652801 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.652656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zpstz" Jul 10 17:38:55.662656 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.662633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:38:55.671345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.671323 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4b7hj" Jul 10 17:38:55.679115 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.679095 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" Jul 10 17:38:55.922152 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.922055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:55.922152 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:55.922110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:55.922381 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.922224 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:55.922381 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.922226 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:55.922381 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.922294 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:56.92227401 +0000 UTC m=+3.983089324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:55.922381 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:55.922311 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:38:56.922302961 +0000 UTC m=+3.983118276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:56.023489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.023455 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:56.023692 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.023665 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:38:56.023763 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.023695 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:38:56.023763 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.023713 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:56.023876 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.023764 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:57.023749572 +0000 UTC m=+4.084564886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:56.341384 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.341295 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:33:55 +0000 UTC" deadline="2028-04-24 01:15:55.973705898 +0000 UTC" Jul 10 17:38:56.341384 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.341323 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15679h36m59.632386269s" Jul 10 17:38:56.427328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.427290 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:56.427500 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.427410 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:38:56.572057 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.572020 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7574bc68_924a_4b78_8734_d00609e93f71.slice/crio-b7071ae9cb82b79394b081c38fa7ffe2bb122be810ce9baa913b6ebfd018eb5b WatchSource:0}: Error finding container b7071ae9cb82b79394b081c38fa7ffe2bb122be810ce9baa913b6ebfd018eb5b: Status 404 returned error can't find the container with id b7071ae9cb82b79394b081c38fa7ffe2bb122be810ce9baa913b6ebfd018eb5b Jul 10 17:38:56.574103 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.573840 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c38fcda_c9c4_4bd0_85e6_4ce5294b3150.slice/crio-b5e0b6a208d31729772a144da3c46ec386d846f1e415518a9ba078bd87b79f2d WatchSource:0}: Error finding container b5e0b6a208d31729772a144da3c46ec386d846f1e415518a9ba078bd87b79f2d: Status 404 returned error can't find the container with id b5e0b6a208d31729772a144da3c46ec386d846f1e415518a9ba078bd87b79f2d Jul 10 17:38:56.575012 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.574676 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f62177_db28_40d7_a0c6_a2aafe07c415.slice/crio-9c25fc783f3b50bd928fbbe63a8ebacaef75b8bf72ad73291e4cc206218476ef WatchSource:0}: Error finding container 9c25fc783f3b50bd928fbbe63a8ebacaef75b8bf72ad73291e4cc206218476ef: Status 404 returned error can't find the container with id 9c25fc783f3b50bd928fbbe63a8ebacaef75b8bf72ad73291e4cc206218476ef Jul 10 17:38:56.579103 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.579046 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a3c326_2280_4cdc_9bb8_1b4fe5a74aa2.slice/crio-51a18d34e1a302b897b29e18c51a3bad61105bb14e9b823c1fcca66d5e69f4ee WatchSource:0}: Error finding container 51a18d34e1a302b897b29e18c51a3bad61105bb14e9b823c1fcca66d5e69f4ee: Status 404 returned error can't find the container with id 51a18d34e1a302b897b29e18c51a3bad61105bb14e9b823c1fcca66d5e69f4ee Jul 10 17:38:56.580555 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.580532 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda092006b_cbfc_4c4a_81f0_a10014c00c2d.slice/crio-be9f4d1784b2f56010b4a6cc056129757791e375cfee319351761bfaf08c394d WatchSource:0}: Error finding container be9f4d1784b2f56010b4a6cc056129757791e375cfee319351761bfaf08c394d: Status 404 returned error can't find the container with id be9f4d1784b2f56010b4a6cc056129757791e375cfee319351761bfaf08c394d Jul 10 17:38:56.585453 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.585401 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfe168c_9893_4b2e_a197_6943e3f9ec2d.slice/crio-dfa81771d2f63605e715db872d9136c7e8b81550a9a67a1be7811ad8371dd58a WatchSource:0}: Error finding container dfa81771d2f63605e715db872d9136c7e8b81550a9a67a1be7811ad8371dd58a: Status 404 returned error can't find the container with id dfa81771d2f63605e715db872d9136c7e8b81550a9a67a1be7811ad8371dd58a Jul 10 17:38:56.586141 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.586007 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c2a170_d162_4a91_959f_5dd6e133855c.slice/crio-5394c47f8fed9d788ce3e986916410f8135a942f0d6260dba3884a150b76fd2d WatchSource:0}: Error finding container 5394c47f8fed9d788ce3e986916410f8135a942f0d6260dba3884a150b76fd2d: Status 404 returned error can't find the container with id 5394c47f8fed9d788ce3e986916410f8135a942f0d6260dba3884a150b76fd2d Jul 10 17:38:56.589440 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:38:56.588199 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480f4dd5_3168_4d70_bfd4_630cc225498a.slice/crio-2d759cfdb7f9840f6e7b61812f78a38c61b0559385c7e058200b0c24468610b2 WatchSource:0}: Error finding container 2d759cfdb7f9840f6e7b61812f78a38c61b0559385c7e058200b0c24468610b2: Status 404 returned error can't find the container with id 2d759cfdb7f9840f6e7b61812f78a38c61b0559385c7e058200b0c24468610b2 Jul 10 17:38:56.932130 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.931930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:56.932259 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:56.932149 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:56.932259 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.932067 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:56.932335 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.932261 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:38:58.932245573 +0000 UTC m=+5.993060888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:56.932335 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.932276 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:56.932335 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:56.932323 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:58.932312036 +0000 UTC m=+5.993127351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:57.032624 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.032588 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:57.032805 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.032759 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:38:57.032805 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.032779 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:38:57.032805 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.032795 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:57.032982 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.032856 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:38:59.032835425 +0000 UTC m=+6.093650747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:57.342276 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.342120 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 17:33:55 +0000 UTC" deadline="2028-03-13 12:47:50.9597784 +0000 UTC" Jul 10 17:38:57.342276 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.342190 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14683h8m53.617593724s" Jul 10 17:38:57.431963 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.431000 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:57.431963 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.431168 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:38:57.431963 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.431717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:57.431963 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:57.431811 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:38:57.448428 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.448393 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qq7vv" event={"ID":"49c2a170-d162-4a91-959f-5dd6e133855c","Type":"ContainerStarted","Data":"5394c47f8fed9d788ce3e986916410f8135a942f0d6260dba3884a150b76fd2d"} Jul 10 17:38:57.463282 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.463247 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lgddx" event={"ID":"1bfe168c-9893-4b2e-a197-6943e3f9ec2d","Type":"ContainerStarted","Data":"dfa81771d2f63605e715db872d9136c7e8b81550a9a67a1be7811ad8371dd58a"} Jul 10 17:38:57.477440 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.477313 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"be9f4d1784b2f56010b4a6cc056129757791e375cfee319351761bfaf08c394d"} Jul 10 17:38:57.484680 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.484591 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerStarted","Data":"51a18d34e1a302b897b29e18c51a3bad61105bb14e9b823c1fcca66d5e69f4ee"} Jul 10 17:38:57.486573 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.486546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6xr2w" event={"ID":"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150","Type":"ContainerStarted","Data":"b5e0b6a208d31729772a144da3c46ec386d846f1e415518a9ba078bd87b79f2d"} Jul 10 17:38:57.495794 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.495764 2582 generic.go:356] "Generic (PLEG): container finished" podID="c7494cf2ba7496408a828f6dd070ca22" containerID="49cf94ab6072767f3f404241ccbde71b233450658990e6294e78543a413758e7" exitCode=0 Jul 10 17:38:57.496220 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.495960 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"c7494cf2ba7496408a828f6dd070ca22","Type":"ContainerDied","Data":"49cf94ab6072767f3f404241ccbde71b233450658990e6294e78543a413758e7"} Jul 10 17:38:57.508112 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.508060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" event={"ID":"3a736e251cd3abf42f7ab6349dd18630","Type":"ContainerStarted","Data":"9614ce3b2229084a0ef596cf19b56d1343ea3418248f6723e772eb728d89f99b"} Jul 10 17:38:57.527952 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.527908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" event={"ID":"f55360d7-4733-4c29-9eed-31ebb15baeaa","Type":"ContainerStarted","Data":"7d4bc756284cb593293c9e7246d4081c9f01cf962139551ab9a04858011e74c5"} Jul 10 17:38:57.529844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.529760 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" podStartSLOduration=2.529741205 podStartE2EDuration="2.529741205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:38:57.529164847 +0000 UTC m=+4.589980188" watchObservedRunningTime="2026-07-10 17:38:57.529741205 +0000 UTC m=+4.590556546" Jul 10 17:38:57.541530 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.541497 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4b7hj" event={"ID":"35f62177-db28-40d7-a0c6-a2aafe07c415","Type":"ContainerStarted","Data":"9c25fc783f3b50bd928fbbe63a8ebacaef75b8bf72ad73291e4cc206218476ef"} Jul 10 17:38:57.554667 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.554595 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zpstz" event={"ID":"7574bc68-924a-4b78-8734-d00609e93f71","Type":"ContainerStarted","Data":"b7071ae9cb82b79394b081c38fa7ffe2bb122be810ce9baa913b6ebfd018eb5b"} Jul 10 17:38:57.565283 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:57.565209 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fr257" event={"ID":"480f4dd5-3168-4d70-bfd4-630cc225498a","Type":"ContainerStarted","Data":"2d759cfdb7f9840f6e7b61812f78a38c61b0559385c7e058200b0c24468610b2"} Jul 10 17:38:58.427429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:58.427376 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:58.427973 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:58.427566 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:38:58.593122 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:58.593084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"c7494cf2ba7496408a828f6dd070ca22","Type":"ContainerStarted","Data":"285a260c821501a1934293739845fa5c530cc9307b6c001b6ee73d4beb72c487"} Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:58.950887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:58.950926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:58.951048 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:58.951123 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:02.951095289 +0000 UTC m=+10.011910624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:58.951194 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:58.951537 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:58.951228 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:39:02.951216421 +0000 UTC m=+10.012031739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:38:59.052826 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:59.052133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:59.052826 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.052316 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:38:59.052826 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.052335 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:38:59.052826 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.052349 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:59.052826 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.052409 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:03.052389296 +0000 UTC m=+10.113204614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:38:59.428291 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:59.428152 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:38:59.428752 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.428338 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:38:59.428752 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:38:59.428702 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:38:59.428843 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:38:59.428812 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:00.427526 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:00.427485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:00.427709 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:00.427635 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:01.429697 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:01.427113 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:01.429697 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:01.427257 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:01.429697 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:01.428066 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:01.429697 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:01.428185 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:02.427093 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:02.427059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:02.427292 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:02.427216 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:02.988748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:02.988706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:02.989259 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:02.988764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:02.989259 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:02.988893 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:02.989259 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:02.988965 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:39:10.988946971 +0000 UTC m=+18.049762287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:02.989259 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:02.988896 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:02.989259 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:02.989079 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:10.989062677 +0000 UTC m=+18.049877996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:03.089263 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:03.089113 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:03.089458 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.089340 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:03.089458 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.089362 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:03.089458 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.089374 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:03.089458 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.089436 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:11.089416188 +0000 UTC m=+18.150231503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:03.427807 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:03.427730 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:03.427955 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.427847 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:03.427955 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:03.427934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:03.428096 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:03.428057 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:04.428030 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:04.427539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:04.428030 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:04.427673 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:05.430387 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:05.430354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:05.430766 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:05.430457 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:05.430766 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:05.430756 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:05.430842 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:05.430817 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:06.426704 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:06.426666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:06.426900 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:06.426795 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:07.430073 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:07.430045 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:07.430541 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:07.430080 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:07.430541 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:07.430175 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:07.430541 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:07.430308 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:08.427071 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:08.427039 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:08.427215 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:08.427162 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:09.426832 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:09.426793 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:09.427313 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:09.426924 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:09.427313 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:09.426985 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:09.427313 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:09.427111 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:10.427457 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:10.427425 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:10.427846 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:10.427544 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:11.053046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:11.053102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.053202 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.053217 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.053272 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:27.053257793 +0000 UTC m=+34.114073105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:11.053321 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.053292 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:39:27.053283122 +0000 UTC m=+34.114098456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:11.153785 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:11.153750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:11.153982 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.153893 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:11.153982 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.153910 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:11.153982 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.153919 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:11.153982 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.153971 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:27.15395258 +0000 UTC m=+34.214767893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:11.427577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:11.427547 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:11.428018 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.427689 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:11.428018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:11.427719 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:11.428018 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:11.427836 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:12.427579 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:12.427543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:12.428067 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:12.427667 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:13.427738 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:13.427703 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:13.428197 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:13.427812 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:13.428197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:13.427869 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:13.428197 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:13.428005 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:14.427358 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:14.427313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:14.427542 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:14.427458 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:15.430907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:15.430879 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:15.430907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:15.430905 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:15.431620 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:15.431034 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:15.431620 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:15.431177 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:16.427492 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.427166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:16.427631 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:16.427611 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:16.633416 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.633371 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lgddx" event={"ID":"1bfe168c-9893-4b2e-a197-6943e3f9ec2d","Type":"ContainerStarted","Data":"9a7a95eaa85c4d839f166d226653d0439562b82124103e9600221339114029a1"} Jul 10 17:39:16.636161 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636142 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:39:16.636508 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636486 2582 generic.go:356] "Generic (PLEG): container finished" podID="a092006b-cbfc-4c4a-81f0-a10014c00c2d" containerID="e19f2f148b62aeb9273ea3a2a2eea3d1f2d302f50b80e47a2bdb5bb9146bd90e" exitCode=1 Jul 10 17:39:16.636575 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636555 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"a1f3f7c7f051f8e69e75ad06da0aa3e57d87a465a9e8a1568b5f6c5d8b43acae"} Jul 10 17:39:16.636641 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636620 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"5706ad7e24c49c98250155b918f9572de104065e8c49e131da8cc1c8f9478ed4"} Jul 10 17:39:16.636765 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"7588dd2e9a70384ef5be13d9617b1ea8ef0d742ddcec8d9f0444666222f61a34"} Jul 10 17:39:16.636765 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636706 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"c31ae7ccf8831ed1d710000b3a16bb27e1cc3e38f8f25d6bfaaa94c45abeefdf"} Jul 10 17:39:16.636765 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636751 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerDied","Data":"e19f2f148b62aeb9273ea3a2a2eea3d1f2d302f50b80e47a2bdb5bb9146bd90e"} Jul 10 17:39:16.637066 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.636799 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"342dcc1ed88e309a693534723d81b788cac408d4da222fa32c4c41da34fb40c2"} Jul 10 17:39:16.637921 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.637899 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="1f6316e1b14a68782072ec33aea97722785b334f0e40eb31a24427483c253552" exitCode=0 Jul 10 17:39:16.638004 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.637967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"1f6316e1b14a68782072ec33aea97722785b334f0e40eb31a24427483c253552"} Jul 10 17:39:16.639416 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.639387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6xr2w" event={"ID":"2c38fcda-c9c4-4bd0-85e6-4ce5294b3150","Type":"ContainerStarted","Data":"e2dedb01fcecd9c800270a48c88c0a092e70c1e1d87c94e2528190be05d9f6b2"} Jul 10 17:39:16.640899 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.640880 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" event={"ID":"f55360d7-4733-4c29-9eed-31ebb15baeaa","Type":"ContainerStarted","Data":"a6c7bf15ba8c37a9fd9fcca00f2667242eea65ba1e0c65318ec5871059cc9dcd"} Jul 10 17:39:16.642149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.642129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4b7hj" event={"ID":"35f62177-db28-40d7-a0c6-a2aafe07c415","Type":"ContainerStarted","Data":"3230184ce257d911113fe4073368a22f2628ddec60d600a16e7672a0d91310b1"} Jul 10 17:39:16.643614 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.643565 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zpstz" event={"ID":"7574bc68-924a-4b78-8734-d00609e93f71","Type":"ContainerStarted","Data":"af7e15a7556ec2cb0086b4e98ff42c63269bb868f4d460982baed6630183152d"} Jul 10 17:39:16.644827 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.644808 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fr257" event={"ID":"480f4dd5-3168-4d70-bfd4-630cc225498a","Type":"ContainerStarted","Data":"2e78a403851c02db2e5d122f476287904c9ccb72ea612f56c2d3ac0981f8eda8"} Jul 10 17:39:16.647334 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.647303 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" podStartSLOduration=21.647292832 podStartE2EDuration="21.647292832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:38:58.610061628 +0000 UTC m=+5.670876968" watchObservedRunningTime="2026-07-10 17:39:16.647292832 +0000 UTC m=+23.708108167" Jul 10 17:39:16.647596 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.647579 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lgddx" podStartSLOduration=11.63926745 podStartE2EDuration="22.64757511s" totalImagesPullingTime="11.00830766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.646467936 +0000 UTC m=+23.707283271" watchObservedRunningTime="2026-07-10 17:39:16.64757511 +0000 UTC m=+23.708390444" Jul 10 17:39:16.662680 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.662647 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6xr2w" podStartSLOduration=3.363265499 podStartE2EDuration="22.662635585s" totalImagesPullingTime="19.299370086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.662405818 +0000 UTC m=+23.723221153" watchObservedRunningTime="2026-07-10 17:39:16.662635585 +0000 UTC m=+23.723450920" Jul 10 17:39:16.691838 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.691794 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fr257" podStartSLOduration=3.327489667 podStartE2EDuration="22.691782425s" totalImagesPullingTime="19.364292758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.677513933 +0000 UTC m=+23.738329269" watchObservedRunningTime="2026-07-10 17:39:16.691782425 +0000 UTC m=+23.752597760" Jul 10 17:39:16.694599 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.692796 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zpstz" podStartSLOduration=3.36092175 podStartE2EDuration="22.692780361s" totalImagesPullingTime="19.331858611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.691910883 +0000 UTC m=+23.752726201" watchObservedRunningTime="2026-07-10 17:39:16.692780361 +0000 UTC m=+23.753595696" Jul 10 17:39:16.725776 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:16.725721 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4b7hj" podStartSLOduration=3.395312321 podStartE2EDuration="22.725701481s" totalImagesPullingTime="19.33038916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:16.725315178 +0000 UTC m=+23.786130513" watchObservedRunningTime="2026-07-10 17:39:16.725701481 +0000 UTC m=+23.786516818" Jul 10 17:39:17.146720 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.146697 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 17:39:17.378211 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.377985 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T17:39:17.146717203Z","UUID":"f004e0b4-b0ea-4fea-8f9c-f12718ed70c8","Handler":null,"Name":"","Endpoint":""} Jul 10 17:39:17.379637 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.379608 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 17:39:17.379753 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.379656 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 17:39:17.431246 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.431221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:17.431438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.431219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:17.431438 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:17.431321 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:17.431438 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:17.431412 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:17.648577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.648492 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" event={"ID":"f55360d7-4733-4c29-9eed-31ebb15baeaa","Type":"ContainerStarted","Data":"1f69c410bcf152da4ebb49a8789282241f96b393b34021354e9c5d4c8b734cdd"} Jul 10 17:39:17.649936 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.649907 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qq7vv" event={"ID":"49c2a170-d162-4a91-959f-5dd6e133855c","Type":"ContainerStarted","Data":"5490241964f64ab5696a8f5e92c6b59304ca4c95932595cfbc163956937394d0"} Jul 10 17:39:17.663197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.663158 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qq7vv" podStartSLOduration=4.346532946 podStartE2EDuration="23.663146246s" totalImagesPullingTime="19.3166133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:17.663033346 +0000 UTC m=+24.723848681" watchObservedRunningTime="2026-07-10 17:39:17.663146246 +0000 UTC m=+24.723961581" Jul 10 17:39:17.836537 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.836509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:39:17.837144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:17.837119 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:39:18.139382 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.139345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:18.249017 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.248967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:18.426557 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.426526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:18.426697 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:18.426622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:18.656581 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.656416 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:39:18.657028 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.656982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"4e21ee6d16681d51f9afc206c06581825437fe6cc9e0615d4a94099f8bfbc792"} Jul 10 17:39:18.659417 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.659380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" event={"ID":"f55360d7-4733-4c29-9eed-31ebb15baeaa","Type":"ContainerStarted","Data":"3c7af3d6f19fa1a22e6bd39878375ada64fe5a5c446dc348f4a20d3dbe192da8"} Jul 10 17:39:18.659888 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.659838 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:39:18.660226 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.660210 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lgddx" Jul 10 17:39:18.678697 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.678613 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msbvm" podStartSLOduration=3.553429967 podStartE2EDuration="24.678601201s" totalImagesPullingTime="21.125171234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:18.678369221 +0000 UTC m=+25.739184558" watchObservedRunningTime="2026-07-10 17:39:18.678601201 +0000 UTC m=+25.739416535" Jul 10 17:39:18.738588 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:18.738551 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:19.430404 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:19.430374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:19.430574 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:19.430374 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:19.430574 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:19.430524 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:19.430668 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:19.430578 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:20.427003 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:20.426955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:20.427463 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:20.427072 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:21.427908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.427677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:21.428696 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:21.428034 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:21.428696 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.428447 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:21.428696 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:21.428540 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:21.670551 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.670520 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:39:21.670933 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.670908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"3d33644881f55675316d0bf87b4ee8ba8de3326b4081672bacf60863724ddc2f"} Jul 10 17:39:21.671225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.671199 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:21.671316 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.671286 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:21.671369 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.671337 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:21.671449 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.671431 2582 scope.go:117] "RemoveContainer" containerID="e19f2f148b62aeb9273ea3a2a2eea3d1f2d302f50b80e47a2bdb5bb9146bd90e" Jul 10 17:39:21.692825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.692737 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:21.693174 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:21.693113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:22.427444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.427421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:22.427568 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:22.427547 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:22.676921 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.676904 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:39:22.677302 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.677277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" event={"ID":"a092006b-cbfc-4c4a-81f0-a10014c00c2d","Type":"ContainerStarted","Data":"56821029785ac905054b7dae48e3a049bb4f967fae44a9274fc0c898128b8a62"} Jul 10 17:39:22.705696 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.705494 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gt5xj"] Jul 10 17:39:22.705812 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.705795 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:22.705896 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:22.705879 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:22.708847 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.708805 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gwzp8"] Jul 10 17:39:22.709150 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.708916 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:22.709150 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:22.709028 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:22.709714 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.709689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wxtvv"] Jul 10 17:39:22.709808 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.709728 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" podStartSLOduration=9.296729443 podStartE2EDuration="28.709709976s" totalImagesPullingTime="19.412980533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:22.708925658 +0000 UTC m=+29.769741035" watchObservedRunningTime="2026-07-10 17:39:22.709709976 +0000 UTC m=+29.770525312" Jul 10 17:39:22.710263 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:22.709818 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:22.710263 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:22.710225 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:23.680660 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:23.680626 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="90209fa426f66dc282f06246cb201ad03a1f78fce6b8eef67abe4de4de5af9dc" exitCode=0 Jul 10 17:39:23.681084 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:23.680727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"90209fa426f66dc282f06246cb201ad03a1f78fce6b8eef67abe4de4de5af9dc"} Jul 10 17:39:23.881806 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:23.881769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:23.937907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:23.937806 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:24.427412 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:24.427375 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:24.427601 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:24.427395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:24.427601 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:24.427503 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:24.427705 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:24.427594 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:24.427705 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:24.427395 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:24.427809 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:24.427708 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:25.089392 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:25.089365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:25.219725 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:25.219690 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:25.686290 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:25.686252 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="26c5aff50aa3528fdbc28cb30af4fcf92abd07e2066c501e0b5d14ceda52a0cb" exitCode=0 Jul 10 17:39:25.686290 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:25.686296 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"26c5aff50aa3528fdbc28cb30af4fcf92abd07e2066c501e0b5d14ceda52a0cb"} Jul 10 17:39:26.427012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:26.426967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:26.427502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:26.427054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:26.427502 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:26.427151 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:26.427502 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:26.427155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:26.427502 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:26.427222 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:26.427502 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:26.427276 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:27.075363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:27.075332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:27.075363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:27.075368 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:27.075529 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.075475 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:27.075529 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.075477 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:27.075529 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.075525 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:59.075512396 +0000 UTC m=+66.136327709 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 17:39:27.075639 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.075537 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret podName:542ae712-d1af-4d6d-99ee-60aeb92da71f nodeName:}" failed. No retries permitted until 2026-07-10 17:39:59.075531673 +0000 UTC m=+66.136346986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret") pod "global-pull-secret-syncer-gwzp8" (UID: "542ae712-d1af-4d6d-99ee-60aeb92da71f") : object "kube-system"/"original-pull-secret" not registered Jul 10 17:39:27.176532 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:27.176328 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:27.176760 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.176531 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 17:39:27.176760 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.176560 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 17:39:27.176760 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.176574 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hdmq9 for pod openshift-network-diagnostics/network-check-target-gt5xj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:27.176760 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:27.176633 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9 podName:7ce31e1b-f96d-47c3-80e1-d910ad806f06 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:59.17661712 +0000 UTC m=+66.237432450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdmq9" (UniqueName: "kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9") pod "network-check-target-gt5xj" (UID: "7ce31e1b-f96d-47c3-80e1-d910ad806f06") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 17:39:27.693414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:27.693385 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="c8ec83f669714eb355733c14770593be7c1a6b3b1d262dfad81bf03586b895cb" exitCode=0 Jul 10 17:39:27.693880 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:27.693432 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"c8ec83f669714eb355733c14770593be7c1a6b3b1d262dfad81bf03586b895cb"} Jul 10 17:39:28.427090 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.427061 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:28.427274 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.427116 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:28.427274 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:28.427209 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gwzp8" podUID="542ae712-d1af-4d6d-99ee-60aeb92da71f" Jul 10 17:39:28.427274 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.427238 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:28.427412 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:28.427326 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:39:28.427412 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:28.427360 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gt5xj" podUID="7ce31e1b-f96d-47c3-80e1-d910ad806f06" Jul 10 17:39:28.809666 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.809600 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeReady" Jul 10 17:39:28.810022 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.809731 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 17:39:28.851603 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.851571 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nzwqh"] Jul 10 17:39:28.871017 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.870971 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7hx"] Jul 10 17:39:28.871174 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.871096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:28.873346 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.873305 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 17:39:28.873605 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.873586 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 17:39:28.873605 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.873598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dbkj9\"" Jul 10 17:39:28.886539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.886511 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nzwqh"] Jul 10 17:39:28.886648 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.886549 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7hx"] Jul 10 17:39:28.886648 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.886631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:28.889018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.888912 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 17:39:28.889018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.888981 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 17:39:28.889198 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.888982 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h8cns\"" Jul 10 17:39:28.889198 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.888913 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 17:39:28.991196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991156 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:28.991366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj92r\" (UniqueName: \"kubernetes.io/projected/88201d50-d8af-4155-85d3-21a6c2c9f47a-kube-api-access-vj92r\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:28.991366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d91d859f-8d57-458c-8d15-2c124ece69c7-config-volume\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:28.991366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88201d50-d8af-4155-85d3-21a6c2c9f47a-cert\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:28.991366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d859f-8d57-458c-8d15-2c124ece69c7-tmp-dir\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:28.991366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:28.991326 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcff\" (UniqueName: \"kubernetes.io/projected/d91d859f-8d57-458c-8d15-2c124ece69c7-kube-api-access-qmcff\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.092647 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d91d859f-8d57-458c-8d15-2c124ece69c7-config-volume\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.092647 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88201d50-d8af-4155-85d3-21a6c2c9f47a-cert\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:29.092858 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d859f-8d57-458c-8d15-2c124ece69c7-tmp-dir\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.092858 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcff\" (UniqueName: \"kubernetes.io/projected/d91d859f-8d57-458c-8d15-2c124ece69c7-kube-api-access-qmcff\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.092981 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.092981 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.092937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj92r\" (UniqueName: \"kubernetes.io/projected/88201d50-d8af-4155-85d3-21a6c2c9f47a-kube-api-access-vj92r\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:29.093115 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:29.093082 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:29.093115 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.093085 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91d859f-8d57-458c-8d15-2c124ece69c7-tmp-dir\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.093242 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:29.093138 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:29.593120841 +0000 UTC m=+36.653936170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:29.093307 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.093243 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d91d859f-8d57-458c-8d15-2c124ece69c7-config-volume\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.097221 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.097201 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88201d50-d8af-4155-85d3-21a6c2c9f47a-cert\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:29.103729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.103705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj92r\" (UniqueName: \"kubernetes.io/projected/88201d50-d8af-4155-85d3-21a6c2c9f47a-kube-api-access-vj92r\") pod \"ingress-canary-tz7hx\" (UID: \"88201d50-d8af-4155-85d3-21a6c2c9f47a\") " pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:29.103865 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.103845 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcff\" (UniqueName: \"kubernetes.io/projected/d91d859f-8d57-458c-8d15-2c124ece69c7-kube-api-access-qmcff\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.133903 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.133873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:29.185727 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.185696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:29.196219 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.196200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tz7hx" Jul 10 17:39:29.405235 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.405201 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tz7hx"] Jul 10 17:39:29.409178 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:39:29.409150 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88201d50_d8af_4155_85d3_21a6c2c9f47a.slice/crio-36ea8a9db46adcb0785e0f5b40e28295398911ec778514a0a4c0c95b30354537 WatchSource:0}: Error finding container 36ea8a9db46adcb0785e0f5b40e28295398911ec778514a0a4c0c95b30354537: Status 404 returned error can't find the container with id 36ea8a9db46adcb0785e0f5b40e28295398911ec778514a0a4c0c95b30354537 Jul 10 17:39:29.598125 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.598086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:29.598289 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:29.598235 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:29.598331 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:29.598296 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:30.598279182 +0000 UTC m=+37.659094497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:29.698474 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:29.698422 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tz7hx" event={"ID":"88201d50-d8af-4155-85d3-21a6c2c9f47a","Type":"ContainerStarted","Data":"36ea8a9db46adcb0785e0f5b40e28295398911ec778514a0a4c0c95b30354537"} Jul 10 17:39:30.427058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.427020 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:30.427537 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.427097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:30.427613 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.427555 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:30.430243 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.430212 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:39:30.431066 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.431043 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:39:30.431164 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.431047 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:39:30.431233 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.431218 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rxjk8\"" Jul 10 17:39:30.431363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.431349 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n82vp\"" Jul 10 17:39:30.431425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.431411 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:39:30.603293 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:30.603257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:30.603476 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:30.603399 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:30.603540 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:30.603478 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:32.603460746 +0000 UTC m=+39.664276080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:32.620412 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:32.620232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:32.620412 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:32.620393 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:32.620876 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:32.620488 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:36.620465471 +0000 UTC m=+43.681280805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:33.635653 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.635414 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:33.690330 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.690301 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:33.711937 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.711912 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="d174fc68ed4014c2ed5022aeade3ac18afac321b10fda98cdfb1139f5d489d37" exitCode=0 Jul 10 17:39:33.712089 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.711982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"d174fc68ed4014c2ed5022aeade3ac18afac321b10fda98cdfb1139f5d489d37"} Jul 10 17:39:33.713366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.713344 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tz7hx" event={"ID":"88201d50-d8af-4155-85d3-21a6c2c9f47a","Type":"ContainerStarted","Data":"6e7c3c0e0aa2a6c2dfb3df9fedcd37c25f48f8e4c59fd7f0d9335779b179e921"} Jul 10 17:39:33.748029 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.747966 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tz7hx" podStartSLOduration=1.747145443 podStartE2EDuration="5.747947451s" totalImagesPullingTime="4.000802008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:33.747104188 +0000 UTC m=+40.807919522" watchObservedRunningTime="2026-07-10 17:39:33.747947451 +0000 UTC m=+40.808762765" Jul 10 17:39:33.758767 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.758700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:33.866728 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.866700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:33.968082 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:33.967980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:34.594957 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:34.594925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:34.718250 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:34.718216 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2" containerID="b0c85bda94dccd23f3be79c3b9fa3534ce7827bcb277935252ce779cce42a9ef" exitCode=0 Jul 10 17:39:34.718748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:34.718305 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerDied","Data":"b0c85bda94dccd23f3be79c3b9fa3534ce7827bcb277935252ce779cce42a9ef"} Jul 10 17:39:34.883148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:34.883086 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:34.934128 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:34.934098 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:35.636728 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.636693 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:35.688250 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.688193 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:35.723825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.723792 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-72p97" event={"ID":"d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2","Type":"ContainerStarted","Data":"6063f5e695076aaa6edafbc49ea00f97f030d2d40259d294e283ef0f683d8a81"} Jul 10 17:39:35.744812 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.744754 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-72p97" podStartSLOduration=8.386008495 podStartE2EDuration="41.74470383s" totalImagesPullingTime="33.358695335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:39:35.74286044 +0000 UTC m=+42.803675785" watchObservedRunningTime="2026-07-10 17:39:35.74470383 +0000 UTC m=+42.805519166" Jul 10 17:39:35.863729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.863696 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:35.924484 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:35.924443 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:36.649277 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:36.649231 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:36.649459 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:36.649387 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:36.649459 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:36.649450 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:39:44.649434224 +0000 UTC m=+51.710249536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:44.701681 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:44.701644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:39:44.702188 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:44.701763 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:39:44.702188 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:44.701836 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:00.701815231 +0000 UTC m=+67.762630548 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:39:45.081623 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:45.081537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:45.131889 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:45.131860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:46.176926 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:46.176892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:46.278451 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:46.278421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:48.476644 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:48.476610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:51.730260 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:51.730226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:51.783963 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:51.783936 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:53.701896 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:53.701869 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-phwsw" Jul 10 17:39:53.852609 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:53.852572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:53.902765 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:53.902739 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:39:59.098656 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.098612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:59.098656 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.098659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:39:59.101058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.101037 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 17:39:59.101177 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.101145 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 17:39:59.108985 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:59.108966 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 17:39:59.109057 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:39:59.109045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:41:03.109029307 +0000 UTC m=+130.169844620 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : secret "metrics-daemon-secret" not found Jul 10 17:39:59.112257 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.112235 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/542ae712-d1af-4d6d-99ee-60aeb92da71f-original-pull-secret\") pod \"global-pull-secret-syncer-gwzp8\" (UID: \"542ae712-d1af-4d6d-99ee-60aeb92da71f\") " pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:59.151222 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.151196 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc"] Jul 10 17:39:59.154896 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.154883 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.157568 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.157539 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 17:39:59.157719 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.157697 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 17:39:59.157790 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.157721 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 17:39:59.158587 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.158563 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 17:39:59.158759 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.158737 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 17:39:59.158885 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.158861 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 17:39:59.159099 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.158683 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 17:39:59.164010 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.163975 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc"] Jul 10 17:39:59.199378 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199354 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:59.199481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.199481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65642de5-b69a-4a90-90fe-f02e17713481-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.199481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199444 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97nf\" (UniqueName: \"kubernetes.io/projected/65642de5-b69a-4a90-90fe-f02e17713481-kube-api-access-g97nf\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.199595 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.199595 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.199595 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.199538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.201435 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.201418 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 17:39:59.211705 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.211691 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 17:39:59.222564 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.222545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmq9\" (UniqueName: \"kubernetes.io/projected/7ce31e1b-f96d-47c3-80e1-d910ad806f06-kube-api-access-hdmq9\") pod \"network-check-target-gt5xj\" (UID: \"7ce31e1b-f96d-47c3-80e1-d910ad806f06\") " pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:59.243344 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.243326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n82vp\"" Jul 10 17:39:59.252183 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.252170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:39:59.255985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.255972 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gwzp8" Jul 10 17:39:59.299904 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.299868 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.300069 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.299918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.300069 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.299946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.301242 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.300639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.301242 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.300770 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65642de5-b69a-4a90-90fe-f02e17713481-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.301242 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.300844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97nf\" (UniqueName: \"kubernetes.io/projected/65642de5-b69a-4a90-90fe-f02e17713481-kube-api-access-g97nf\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.301747 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.301685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65642de5-b69a-4a90-90fe-f02e17713481-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.303309 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.303278 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.303966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.303687 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.306267 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.306050 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.306267 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.306199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65642de5-b69a-4a90-90fe-f02e17713481-ca\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.315515 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.315459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97nf\" (UniqueName: \"kubernetes.io/projected/65642de5-b69a-4a90-90fe-f02e17713481-kube-api-access-g97nf\") pod \"cluster-proxy-proxy-agent-7ddd5776bc-hvvhc\" (UID: \"65642de5-b69a-4a90-90fe-f02e17713481\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.407394 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.407365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gwzp8"] Jul 10 17:39:59.410514 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:39:59.410486 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542ae712_d1af_4d6d_99ee_60aeb92da71f.slice/crio-3e94a9403cb7d21246d99478d8d0a3159aaa078ae91b285f145e3472f706e2ee WatchSource:0}: Error finding container 3e94a9403cb7d21246d99478d8d0a3159aaa078ae91b285f145e3472f706e2ee: Status 404 returned error can't find the container with id 3e94a9403cb7d21246d99478d8d0a3159aaa078ae91b285f145e3472f706e2ee Jul 10 17:39:59.423149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.423126 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gt5xj"] Jul 10 17:39:59.426111 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:39:59.426091 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce31e1b_f96d_47c3_80e1_d910ad806f06.slice/crio-3cbc4fe85bc85afeee89e602fc4e225d154bca2eba90d1dfe66e857821d758ad WatchSource:0}: Error finding container 3cbc4fe85bc85afeee89e602fc4e225d154bca2eba90d1dfe66e857821d758ad: Status 404 returned error can't find the container with id 3cbc4fe85bc85afeee89e602fc4e225d154bca2eba90d1dfe66e857821d758ad Jul 10 17:39:59.478756 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.478732 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:39:59.595267 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.595238 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc"] Jul 10 17:39:59.597661 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:39:59.597632 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65642de5_b69a_4a90_90fe_f02e17713481.slice/crio-ffd7d3a33690eaa681a29fb61b43aeaff89efaaffa725b28a82787016b56e7e9 WatchSource:0}: Error finding container ffd7d3a33690eaa681a29fb61b43aeaff89efaaffa725b28a82787016b56e7e9: Status 404 returned error can't find the container with id ffd7d3a33690eaa681a29fb61b43aeaff89efaaffa725b28a82787016b56e7e9 Jul 10 17:39:59.776881 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.776850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gwzp8" event={"ID":"542ae712-d1af-4d6d-99ee-60aeb92da71f","Type":"ContainerStarted","Data":"3e94a9403cb7d21246d99478d8d0a3159aaa078ae91b285f145e3472f706e2ee"} Jul 10 17:39:59.777875 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.777853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gt5xj" event={"ID":"7ce31e1b-f96d-47c3-80e1-d910ad806f06","Type":"ContainerStarted","Data":"3cbc4fe85bc85afeee89e602fc4e225d154bca2eba90d1dfe66e857821d758ad"} Jul 10 17:39:59.778971 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:39:59.778947 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerStarted","Data":"ffd7d3a33690eaa681a29fb61b43aeaff89efaaffa725b28a82787016b56e7e9"} Jul 10 17:40:00.710046 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:00.710005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:40:00.710517 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:40:00.710220 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:40:00.710517 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:40:00.710281 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:40:32.71026272 +0000 UTC m=+99.771078038 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:40:04.795488 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:04.795452 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gt5xj" event={"ID":"7ce31e1b-f96d-47c3-80e1-d910ad806f06","Type":"ContainerStarted","Data":"da9206f38df6cb313c69855a96e29a06d3652688a8a004aa428c7d2cc30506b0"} Jul 10 17:40:04.795970 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:04.795542 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:40:04.797087 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:04.797060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gwzp8" event={"ID":"542ae712-d1af-4d6d-99ee-60aeb92da71f","Type":"ContainerStarted","Data":"00baa10e33f15faf66219d30b9514cad8a08f026e82648aa52317655507037dc"} Jul 10 17:40:04.810105 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:04.810065 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gt5xj" podStartSLOduration=65.658125842 podStartE2EDuration="1m10.810052136s" totalImagesPullingTime="5.151926294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:04.809550009 +0000 UTC m=+71.870365343" watchObservedRunningTime="2026-07-10 17:40:04.810052136 +0000 UTC m=+71.870867470" Jul 10 17:40:04.823028 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:04.822966 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gwzp8" podStartSLOduration=65.651269891 podStartE2EDuration="1m10.822951932s" totalImagesPullingTime="5.171682041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:04.821777652 +0000 UTC m=+71.882592986" watchObservedRunningTime="2026-07-10 17:40:04.822951932 +0000 UTC m=+71.883767266" Jul 10 17:40:10.814233 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:10.814200 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerStarted","Data":"056388fcfb2e5bb0544c7b00b4ebec826419f669a121d8e2b46b08e78f754277"} Jul 10 17:40:10.814589 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:10.814249 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerStarted","Data":"e6ebe40bc19dbe87316b38d70a3466dd54ca1c7b3b10cbc758ce5fcc429e3eb1"} Jul 10 17:40:10.814589 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:10.814266 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerStarted","Data":"f2db73f1c3bb631fa667091bcf89567cd49734884772682d5f69aa13487f9a6b"} Jul 10 17:40:10.830863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:10.830822 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" podStartSLOduration=0.805024531 podStartE2EDuration="11.830809124s" totalImagesPullingTime="11.025784593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:40:10.829957844 +0000 UTC m=+77.890773181" watchObservedRunningTime="2026-07-10 17:40:10.830809124 +0000 UTC m=+77.891624458" Jul 10 17:40:11.662634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:11.662604 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:11.716727 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:11.716692 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:11.770204 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:11.770179 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:40:32.724723 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:32.724692 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:40:32.725136 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:40:32.724819 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:40:32.725136 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:40:32.724886 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:41:36.724870119 +0000 UTC m=+163.785685433 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:40:35.801766 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:40:35.801733 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gt5xj" Jul 10 17:41:03.128193 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:41:03.128130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:41:03.128631 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:03.128296 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 17:41:03.128631 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:03.128368 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:05.128353366 +0000 UTC m=+252.189168678 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : secret "metrics-daemon-secret" not found Jul 10 17:41:31.881338 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:31.881277 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nzwqh" podUID="d91d859f-8d57-458c-8d15-2c124ece69c7" Jul 10 17:41:32.046024 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:41:32.045607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:41:33.449205 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:33.449169 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:41:36.758199 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:41:36.758165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:41:36.758575 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:36.758271 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 17:41:36.758575 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:41:36.758344 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls podName:d91d859f-8d57-458c-8d15-2c124ece69c7 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:38.758327857 +0000 UTC m=+285.819143170 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls") pod "dns-default-nzwqh" (UID: "d91d859f-8d57-458c-8d15-2c124ece69c7") : secret "dns-default-metrics-tls" not found Jul 10 17:41:44.427214 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:41:44.427118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:42:50.170528 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:50.170496 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:59.763246 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.763210 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:42:59.765148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.765127 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx"] Jul 10 17:42:59.765260 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.765241 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.766766 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.766747 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.768523 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.768508 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 17:42:59.769223 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.769207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tc8df\"" Jul 10 17:42:59.770183 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770149 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 17:42:59.770429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770413 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 17:42:59.770519 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770428 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tj6dt\"" Jul 10 17:42:59.770519 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770427 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 17:42:59.770631 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770427 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 17:42:59.770862 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.770844 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 17:42:59.771172 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.771155 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 17:42:59.774855 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.774834 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 17:42:59.781015 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.780974 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx"] Jul 10 17:42:59.781703 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.781688 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:42:59.895564 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895540 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895689 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895689 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895587 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvgx\" (UniqueName: \"kubernetes.io/projected/ba6b12af-bddd-4a83-b719-f11bb49d2eec-kube-api-access-jnvgx\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.895689 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kj5\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6b12af-bddd-4a83-b719-f11bb49d2eec-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.895791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895816 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6b12af-bddd-4a83-b719-f11bb49d2eec-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.895908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.895908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.895869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.901716 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.901689 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:59.953866 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.953841 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:42:59.996965 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.996942 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kj5\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997053 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.996980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997053 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997036 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6b12af-bddd-4a83-b719-f11bb49d2eec-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.997147 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997147 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997147 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6b12af-bddd-4a83-b719-f11bb49d2eec-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997157 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvgx\" (UniqueName: \"kubernetes.io/projected/ba6b12af-bddd-4a83-b719-f11bb49d2eec-kube-api-access-jnvgx\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:42:59.997318 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:42:59.997349 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:42:59.997336 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6744d557b8-864vw: secret "image-registry-tls" not found Jul 10 17:42:59.997640 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:42:59.997402 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls podName:86ee37d7-3eae-488a-baec-557ab007695a nodeName:}" failed. No retries permitted until 2026-07-10 17:43:00.497383197 +0000 UTC m=+247.558198530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls") pod "image-registry-6744d557b8-864vw" (UID: "86ee37d7-3eae-488a-baec-557ab007695a") : secret "image-registry-tls" not found Jul 10 17:42:59.997640 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997491 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997792 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.997976 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.997955 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba6b12af-bddd-4a83-b719-f11bb49d2eec-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:42:59.998206 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.998184 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:42:59.999934 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.999911 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:00.000051 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.999938 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba6b12af-bddd-4a83-b719-f11bb49d2eec-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:43:00.000051 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:42:59.999958 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:00.005800 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.005780 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvgx\" (UniqueName: \"kubernetes.io/projected/ba6b12af-bddd-4a83-b719-f11bb49d2eec-kube-api-access-jnvgx\") pod \"kube-storage-version-migrator-operator-799cb7f958-mzfgx\" (UID: \"ba6b12af-bddd-4a83-b719-f11bb49d2eec\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:43:00.005895 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.005815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kj5\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:00.005895 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.005880 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:00.080157 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.080085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" Jul 10 17:43:00.197434 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.197398 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx"] Jul 10 17:43:00.200832 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:00.200809 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6b12af_bddd_4a83_b719_f11bb49d2eec.slice/crio-33beeef661cb491dd934362e4a2b2f9587792f506d5a9f3c04f3dd1341263a4b WatchSource:0}: Error finding container 33beeef661cb491dd934362e4a2b2f9587792f506d5a9f3c04f3dd1341263a4b: Status 404 returned error can't find the container with id 33beeef661cb491dd934362e4a2b2f9587792f506d5a9f3c04f3dd1341263a4b Jul 10 17:43:00.287547 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.287519 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" event={"ID":"ba6b12af-bddd-4a83-b719-f11bb49d2eec","Type":"ContainerStarted","Data":"33beeef661cb491dd934362e4a2b2f9587792f506d5a9f3c04f3dd1341263a4b"} Jul 10 17:43:00.501577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:00.501547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:00.501741 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:00.501691 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:43:00.501741 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:00.501710 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6744d557b8-864vw: secret "image-registry-tls" not found Jul 10 17:43:00.501826 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:00.501777 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls podName:86ee37d7-3eae-488a-baec-557ab007695a nodeName:}" failed. No retries permitted until 2026-07-10 17:43:01.50176147 +0000 UTC m=+248.562576783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls") pod "image-registry-6744d557b8-864vw" (UID: "86ee37d7-3eae-488a-baec-557ab007695a") : secret "image-registry-tls" not found Jul 10 17:43:01.507898 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:01.507854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:01.508296 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:01.508012 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:43:01.508296 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:01.508030 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6744d557b8-864vw: secret "image-registry-tls" not found Jul 10 17:43:01.508296 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:01.508119 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls podName:86ee37d7-3eae-488a-baec-557ab007695a nodeName:}" failed. No retries permitted until 2026-07-10 17:43:03.508103982 +0000 UTC m=+250.568919298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls") pod "image-registry-6744d557b8-864vw" (UID: "86ee37d7-3eae-488a-baec-557ab007695a") : secret "image-registry-tls" not found Jul 10 17:43:02.297660 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:02.297626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" event={"ID":"ba6b12af-bddd-4a83-b719-f11bb49d2eec","Type":"ContainerStarted","Data":"827b256225b18337843ed09d36175f89438509404edd73983a20b74891206f9f"} Jul 10 17:43:02.311800 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:02.311760 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" podStartSLOduration=1.437556601 podStartE2EDuration="3.311746014s" totalImagesPullingTime="1.874189413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:02.31118011 +0000 UTC m=+249.371995445" watchObservedRunningTime="2026-07-10 17:43:02.311746014 +0000 UTC m=+249.372561349" Jul 10 17:43:03.194043 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.194005 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:03.443064 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.443034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:03.520478 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.520399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:03.520617 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.520513 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:43:03.520617 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.520524 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6744d557b8-864vw: secret "image-registry-tls" not found Jul 10 17:43:03.520617 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.520570 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls podName:86ee37d7-3eae-488a-baec-557ab007695a nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.520556043 +0000 UTC m=+254.581371356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls") pod "image-registry-6744d557b8-864vw" (UID: "86ee37d7-3eae-488a-baec-557ab007695a") : secret "image-registry-tls" not found Jul 10 17:43:03.523458 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.523431 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:03.528122 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.528098 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7fb4r"] Jul 10 17:43:03.529931 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.529917 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.532208 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.532122 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 17:43:03.532208 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.532126 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 17:43:03.532414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.532400 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 17:43:03.533148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.533128 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-66sjw\"" Jul 10 17:43:03.533274 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.533149 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 17:43:03.541782 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.541764 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7fb4r"] Jul 10 17:43:03.587213 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.587187 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:03.621354 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.621331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.621503 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.621386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4pt\" (UniqueName: \"kubernetes.io/projected/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-api-access-7z4pt\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.621503 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.621443 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76af51e2-d96d-4878-af5b-73cc8be7e4d8-data-volume\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.621503 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.621472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76af51e2-d96d-4878-af5b-73cc8be7e4d8-crio-socket\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.621601 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.621518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.721794 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.721772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.721882 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.721807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4pt\" (UniqueName: \"kubernetes.io/projected/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-api-access-7z4pt\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.721882 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.721833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76af51e2-d96d-4878-af5b-73cc8be7e4d8-data-volume\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.721882 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.721870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76af51e2-d96d-4878-af5b-73cc8be7e4d8-crio-socket\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.722028 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.721893 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:03.722028 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.721897 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.722028 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.721946 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:04.221932176 +0000 UTC m=+251.282747492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : configmap "kube-rbac-proxy" not found Jul 10 17:43:03.722028 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.721963 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:03.722187 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:03.722040 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:04.222024477 +0000 UTC m=+251.282839795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:03.722187 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.722056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76af51e2-d96d-4878-af5b-73cc8be7e4d8-crio-socket\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.722297 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.722272 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76af51e2-d96d-4878-af5b-73cc8be7e4d8-data-volume\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:03.731865 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:03.731845 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4pt\" (UniqueName: \"kubernetes.io/projected/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-api-access-7z4pt\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:04.225781 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.225740 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:04.226179 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.225807 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:04.226179 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:04.225896 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:04.226179 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:04.225913 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:04.226179 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:04.225974 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:05.225956913 +0000 UTC m=+252.286772226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : configmap "kube-rbac-proxy" not found Jul 10 17:43:04.226179 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:04.226007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:05.22598176 +0000 UTC m=+252.286797074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:04.276288 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.276258 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:04.332898 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.332868 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:04.475247 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.475216 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:04.576214 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.576134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:04.983727 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:04.983694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:05.084567 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:05.084536 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:05.131601 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:05.131568 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:43:05.131733 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.131715 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 17:43:05.131775 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.131768 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs podName:62f47418-60ee-40ac-83f7-9b62d2a18531 nodeName:}" failed. No retries permitted until 2026-07-10 17:45:07.131753939 +0000 UTC m=+374.192569256 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs") pod "network-metrics-daemon-wxtvv" (UID: "62f47418-60ee-40ac-83f7-9b62d2a18531") : secret "metrics-daemon-secret" not found Jul 10 17:43:05.232077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:05.232049 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:05.232456 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:05.232134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:05.232456 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.232166 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:05.232456 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.232231 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:05.232456 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.232247 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.23223129 +0000 UTC m=+254.293046613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : configmap "kube-rbac-proxy" not found Jul 10 17:43:05.232456 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:05.232270 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:07.23225727 +0000 UTC m=+254.293072583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:05.310530 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:05.310469 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4b7hj_35f62177-db28-40d7-a0c6-a2aafe07c415/dns-node-resolver/0.log" Jul 10 17:43:06.510593 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:06.510568 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6xr2w_2c38fcda-c9c4-4bd0-85e6-4ce5294b3150/node-ca/0.log" Jul 10 17:43:06.681853 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:06.681822 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:06.783489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:06.783410 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:07.248944 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:07.248908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:07.249144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:07.248971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:07.249144 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.249064 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:07.249144 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.249075 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:07.249144 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.249137 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:11.249121541 +0000 UTC m=+258.309936854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:07.249306 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.249150 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:11.249144195 +0000 UTC m=+258.309959507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : configmap "kube-rbac-proxy" not found Jul 10 17:43:07.310298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:07.310264 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tz7hx_88201d50-d8af-4155-85d3-21a6c2c9f47a/serve-healthcheck-canary/0.log" Jul 10 17:43:07.469662 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:07.469628 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:07.551363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:07.551267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:07.551763 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.551404 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 17:43:07.551763 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.551427 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6744d557b8-864vw: secret "image-registry-tls" not found Jul 10 17:43:07.551763 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:07.551507 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls podName:86ee37d7-3eae-488a-baec-557ab007695a nodeName:}" failed. No retries permitted until 2026-07-10 17:43:15.55148512 +0000 UTC m=+262.612300437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls") pod "image-registry-6744d557b8-864vw" (UID: "86ee37d7-3eae-488a-baec-557ab007695a") : secret "image-registry-tls" not found Jul 10 17:43:08.014700 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.014640 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.299309 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.299205 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.402135 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.402103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.544295 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.544250 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.646831 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.646726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.744778 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.744736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:08.800684 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:08.800643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:09.479783 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:09.479738 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" podUID="65642de5-b69a-4a90-90fe-f02e17713481" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 17:43:09.504530 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:09.504498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:09.717581 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:09.717546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:10.110587 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:10.110558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:10.586577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:10.586547 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:10.703909 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:10.703871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:10.918162 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:10.918133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:11.279225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:11.279136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:11.279225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:11.279199 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:11.279428 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:11.279297 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 17:43:11.279428 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:11.279355 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:19.279340901 +0000 UTC m=+266.340156217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : secret "insights-runtime-extractor-tls" not found Jul 10 17:43:11.279428 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:11.279302 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 17:43:11.279428 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:11.279389 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm podName:76af51e2-d96d-4878-af5b-73cc8be7e4d8 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:19.279380801 +0000 UTC m=+266.340196113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fb4r" (UID: "76af51e2-d96d-4878-af5b-73cc8be7e4d8") : configmap "kube-rbac-proxy" not found Jul 10 17:43:11.310514 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:11.310483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:11.906523 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:11.906494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:12.006956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:12.006924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:12.106823 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:12.106786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:12.511420 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:12.511387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:12.704752 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:12.704721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:12.906343 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:12.906295 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:13.106740 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:13.106707 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:13.894975 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:13.894945 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:13.960314 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:13.960284 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:14.305775 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:14.305705 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:14.590665 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:14.590586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:14.709029 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:14.709004 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:15.105193 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.105148 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:15.306538 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.306506 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:15.611383 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.611343 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:15.613862 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.613838 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"image-registry-6744d557b8-864vw\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:15.675073 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.675032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:15.799280 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.799244 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:43:15.801976 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:15.801942 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ee37d7_3eae_488a_baec_557ab007695a.slice/crio-51bc2263daa502dfa049700d042307b237bb7622d746c751769dbb43b2b44e28 WatchSource:0}: Error finding container 51bc2263daa502dfa049700d042307b237bb7622d746c751769dbb43b2b44e28: Status 404 returned error can't find the container with id 51bc2263daa502dfa049700d042307b237bb7622d746c751769dbb43b2b44e28 Jul 10 17:43:15.903063 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:15.903029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:16.337768 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.337689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6744d557b8-864vw" event={"ID":"86ee37d7-3eae-488a-baec-557ab007695a","Type":"ContainerStarted","Data":"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510"} Jul 10 17:43:16.337768 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.337741 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6744d557b8-864vw" event={"ID":"86ee37d7-3eae-488a-baec-557ab007695a","Type":"ContainerStarted","Data":"51bc2263daa502dfa049700d042307b237bb7622d746c751769dbb43b2b44e28"} Jul 10 17:43:16.338163 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.337822 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:16.354872 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.354828 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6744d557b8-864vw" podStartSLOduration=17.354815414 podStartE2EDuration="17.354815414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:16.354559585 +0000 UTC m=+263.415374913" watchObservedRunningTime="2026-07-10 17:43:16.354815414 +0000 UTC m=+263.415630749" Jul 10 17:43:16.502893 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.502862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:16.650927 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.650899 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:16.748682 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.748650 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:16.944024 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:16.943918 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:17.111908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:17.111859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.103334 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.103305 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.302347 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.302319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.503478 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.503448 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.603966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.603934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.703747 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.703715 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:18.755661 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:18.755587 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:19.110649 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.110572 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:19.342506 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.342464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:19.342707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.342532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:19.343095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.343074 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76af51e2-d96d-4878-af5b-73cc8be7e4d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:19.345246 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.345225 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76af51e2-d96d-4878-af5b-73cc8be7e4d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fb4r\" (UID: \"76af51e2-d96d-4878-af5b-73cc8be7e4d8\") " pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:19.439336 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.439310 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7fb4r" Jul 10 17:43:19.476062 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.476029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:19.480195 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.480170 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" podUID="65642de5-b69a-4a90-90fe-f02e17713481" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 17:43:19.553455 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.553427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:19.563615 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:19.563590 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7fb4r"] Jul 10 17:43:19.566777 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:19.566757 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76af51e2_d96d_4878_af5b_73cc8be7e4d8.slice/crio-cdd4934c352d458b1e40bdd2829207d6247728c1e17caa9e82e498b017958bbc WatchSource:0}: Error finding container cdd4934c352d458b1e40bdd2829207d6247728c1e17caa9e82e498b017958bbc: Status 404 returned error can't find the container with id cdd4934c352d458b1e40bdd2829207d6247728c1e17caa9e82e498b017958bbc Jul 10 17:43:20.351455 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:20.351424 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fb4r" event={"ID":"76af51e2-d96d-4878-af5b-73cc8be7e4d8","Type":"ContainerStarted","Data":"cdd4934c352d458b1e40bdd2829207d6247728c1e17caa9e82e498b017958bbc"} Jul 10 17:43:22.027160 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:22.027122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:22.126754 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:22.126700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:22.178149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:22.178121 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:22.316971 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:22.316910 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:22.698187 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:22.698152 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.097114 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.097030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.154119 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.154078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.303345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.303322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.361801 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.361725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fb4r" event={"ID":"76af51e2-d96d-4878-af5b-73cc8be7e4d8","Type":"ContainerStarted","Data":"b122889a6289f1388af9ece1a7d658050c396561594c989b93aa5859bd2cec77"} Jul 10 17:43:23.496368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.496338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.599674 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.599639 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.756087 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.756054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:23.901774 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:23.901747 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:24.367278 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:24.367238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fb4r" event={"ID":"76af51e2-d96d-4878-af5b-73cc8be7e4d8","Type":"ContainerStarted","Data":"85274362dc0251af62690f353fdbbf64278f7ff7e6763dd9266fe832425eb7e8"} Jul 10 17:43:25.124860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:25.124827 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:25.177089 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:25.177042 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:26.375657 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:26.375626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fb4r" event={"ID":"76af51e2-d96d-4878-af5b-73cc8be7e4d8","Type":"ContainerStarted","Data":"bedb30d35330f4de005fc13ba037caa93ba091821b1f60ad5c28c46c9bb26730"} Jul 10 17:43:26.391878 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:26.391839 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7fb4r" podStartSLOduration=16.850647455 podStartE2EDuration="23.391825567s" totalImagesPullingTime="6.541178112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:43:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:26.39125528 +0000 UTC m=+273.452070617" watchObservedRunningTime="2026-07-10 17:43:26.391825567 +0000 UTC m=+273.452640902" Jul 10 17:43:27.679403 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.679366 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:43:27.718247 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.718214 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b8dfdb869-tdt7n"] Jul 10 17:43:27.721141 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.721126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.736084 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.736059 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b8dfdb869-tdt7n"] Jul 10 17:43:27.768005 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.767972 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:27.813612 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-registry-certificates\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-trusted-ca\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7096471e-80b4-4b31-8594-88e19c4951da-ca-trust-extracted\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-installation-pull-secrets\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-registry-tls\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813790 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-image-registry-private-configuration\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-bound-sa-token\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.813961 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.813841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dw8\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-kube-api-access-k8dw8\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.865487 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.865458 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:27.914535 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dw8\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-kube-api-access-k8dw8\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-registry-certificates\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914578 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-trusted-ca\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914596 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7096471e-80b4-4b31-8594-88e19c4951da-ca-trust-extracted\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914633 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914625 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-installation-pull-secrets\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914872 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-registry-tls\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914872 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-image-registry-private-configuration\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.914872 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.914707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-bound-sa-token\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.915087 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.915066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7096471e-80b4-4b31-8594-88e19c4951da-ca-trust-extracted\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.915395 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.915378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-registry-certificates\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.915825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.915804 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7096471e-80b4-4b31-8594-88e19c4951da-trusted-ca\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.917243 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.917219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-image-registry-private-configuration\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.917312 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.917275 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7096471e-80b4-4b31-8594-88e19c4951da-installation-pull-secrets\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.917491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.917470 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-registry-tls\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.925371 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.925345 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-bound-sa-token\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.926323 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.926304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dw8\" (UniqueName: \"kubernetes.io/projected/7096471e-80b4-4b31-8594-88e19c4951da-kube-api-access-k8dw8\") pod \"image-registry-6b8dfdb869-tdt7n\" (UID: \"7096471e-80b4-4b31-8594-88e19c4951da\") " pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:27.965818 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:27.965744 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:28.029082 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.029057 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:28.154319 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.154290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b8dfdb869-tdt7n"] Jul 10 17:43:28.157201 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:28.157173 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7096471e_80b4_4b31_8594_88e19c4951da.slice/crio-36fe3c96e0436cf0dd3ab10b415e44f7088de642f65462f80af74ee5c54538d5 WatchSource:0}: Error finding container 36fe3c96e0436cf0dd3ab10b415e44f7088de642f65462f80af74ee5c54538d5: Status 404 returned error can't find the container with id 36fe3c96e0436cf0dd3ab10b415e44f7088de642f65462f80af74ee5c54538d5 Jul 10 17:43:28.383555 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.383475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" event={"ID":"7096471e-80b4-4b31-8594-88e19c4951da","Type":"ContainerStarted","Data":"c6524daf9d9e0329859bfe50f5d030b091a20a0b756c12762300e477aa2b5181"} Jul 10 17:43:28.383555 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.383523 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" event={"ID":"7096471e-80b4-4b31-8594-88e19c4951da","Type":"ContainerStarted","Data":"36fe3c96e0436cf0dd3ab10b415e44f7088de642f65462f80af74ee5c54538d5"} Jul 10 17:43:28.383739 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.383613 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:28.401004 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:28.400955 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" podStartSLOduration=1.4009403009999999 podStartE2EDuration="1.400940301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:28.400409605 +0000 UTC m=+275.461224939" watchObservedRunningTime="2026-07-10 17:43:28.400940301 +0000 UTC m=+275.461755635" Jul 10 17:43:29.137972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.137944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:29.196634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.196600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:29.480544 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.480511 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" podUID="65642de5-b69a-4a90-90fe-f02e17713481" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 17:43:29.480708 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.480567 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" Jul 10 17:43:29.481133 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.481101 2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"056388fcfb2e5bb0544c7b00b4ebec826419f669a121d8e2b46b08e78f754277"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 17:43:29.481178 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:29.481158 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" podUID="65642de5-b69a-4a90-90fe-f02e17713481" containerName="service-proxy" containerID="cri-o://056388fcfb2e5bb0544c7b00b4ebec826419f669a121d8e2b46b08e78f754277" gracePeriod=30 Jul 10 17:43:30.391430 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:30.391397 2582 generic.go:356] "Generic (PLEG): container finished" podID="65642de5-b69a-4a90-90fe-f02e17713481" containerID="056388fcfb2e5bb0544c7b00b4ebec826419f669a121d8e2b46b08e78f754277" exitCode=2 Jul 10 17:43:30.391842 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:30.391459 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerDied","Data":"056388fcfb2e5bb0544c7b00b4ebec826419f669a121d8e2b46b08e78f754277"} Jul 10 17:43:30.391842 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:30.391491 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7ddd5776bc-hvvhc" event={"ID":"65642de5-b69a-4a90-90fe-f02e17713481","Type":"ContainerStarted","Data":"e1ea7d6f475ae475e0d9ee0baff14bccead5fea733954a6d515fcf07692a3bba"} Jul 10 17:43:30.745774 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:30.745732 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:30.850000 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:30.849952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:31.696540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:31.696505 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:31.763881 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:31.763845 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:31.879886 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:31.879860 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:34.228289 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:34.228248 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:35.046617 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:35.046567 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nzwqh" podUID="d91d859f-8d57-458c-8d15-2c124ece69c7" Jul 10 17:43:35.409230 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:35.409204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:35.830927 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:35.830840 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.049304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.049271 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.102607 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.102542 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2f6c"] Jul 10 17:43:36.108706 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.108675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.112040 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.112019 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 17:43:36.112149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.112126 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 17:43:36.112417 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.112402 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 17:43:36.113088 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.113070 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 17:43:36.113166 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.113108 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 17:43:36.113229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.113178 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lp7qm\"" Jul 10 17:43:36.113270 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.113264 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 17:43:36.138007 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.137966 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.178228 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178232 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-metrics-client-ca\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178297 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-root\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-textfile\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsq4d\" (UniqueName: \"kubernetes.io/projected/ba52d2bc-622c-430e-b437-aa91152f4524-kube-api-access-hsq4d\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178604 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-wtmp\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178604 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-sys\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.178604 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.178525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.199986 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.199961 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.279634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279605 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-root\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-textfile\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279709 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-root\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsq4d\" (UniqueName: \"kubernetes.io/projected/ba52d2bc-622c-430e-b437-aa91152f4524-kube-api-access-hsq4d\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-wtmp\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-sys\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279864 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.279908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279894 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279938 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-sys\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-metrics-client-ca\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.279972 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-textfile\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.280010 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-wtmp\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:36.280101 2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 17:43:36.280304 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:36.280160 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls podName:ba52d2bc-622c-430e-b437-aa91152f4524 nodeName:}" failed. No retries permitted until 2026-07-10 17:43:36.780139837 +0000 UTC m=+283.840955153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls") pod "node-exporter-v2f6c" (UID: "ba52d2bc-622c-430e-b437-aa91152f4524") : secret "node-exporter-tls" not found Jul 10 17:43:36.280623 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.280492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.280623 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.280531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba52d2bc-622c-430e-b437-aa91152f4524-metrics-client-ca\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.282280 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.282251 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.288147 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.288124 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsq4d\" (UniqueName: \"kubernetes.io/projected/ba52d2bc-622c-430e-b437-aa91152f4524-kube-api-access-hsq4d\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.385429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.385344 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.485805 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.485757 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.585555 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.585521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:36.783609 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.783572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.785868 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.785847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba52d2bc-622c-430e-b437-aa91152f4524-node-exporter-tls\") pod \"node-exporter-v2f6c\" (UID: \"ba52d2bc-622c-430e-b437-aa91152f4524\") " pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:36.917414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:36.917347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.018041 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.018015 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2f6c" Jul 10 17:43:37.032604 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:37.032410 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba52d2bc_622c_430e_b437_aa91152f4524.slice/crio-de767c6aa6c25bb86473382c700761daf170374e74fadbfa8b49840faa636de8 WatchSource:0}: Error finding container de767c6aa6c25bb86473382c700761daf170374e74fadbfa8b49840faa636de8: Status 404 returned error can't find the container with id de767c6aa6c25bb86473382c700761daf170374e74fadbfa8b49840faa636de8 Jul 10 17:43:37.058215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.058184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.152508 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.152479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.270066 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.270025 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:37.415462 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.415431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2f6c" event={"ID":"ba52d2bc-622c-430e-b437-aa91152f4524","Type":"ContainerStarted","Data":"de767c6aa6c25bb86473382c700761daf170374e74fadbfa8b49840faa636de8"} Jul 10 17:43:37.684483 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.684407 2582 patch_prober.go:28] interesting pod/image-registry-6744d557b8-864vw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 17:43:37.684483 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:37.684456 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6744d557b8-864vw" podUID="86ee37d7-3eae-488a-baec-557ab007695a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:43:38.419727 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:38.419695 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba52d2bc-622c-430e-b437-aa91152f4524" containerID="20ff1e6961df976726450bace1d847b8392beacdf32f8726501b0ecb591f39a6" exitCode=0 Jul 10 17:43:38.419888 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:38.419738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2f6c" event={"ID":"ba52d2bc-622c-430e-b437-aa91152f4524","Type":"ContainerDied","Data":"20ff1e6961df976726450bace1d847b8392beacdf32f8726501b0ecb591f39a6"} Jul 10 17:43:38.799558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:38.799470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:38.801854 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:38.801833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d91d859f-8d57-458c-8d15-2c124ece69c7-metrics-tls\") pod \"dns-default-nzwqh\" (UID: \"d91d859f-8d57-458c-8d15-2c124ece69c7\") " pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:39.012667 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.012634 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dbkj9\"" Jul 10 17:43:39.020616 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.020591 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:39.050373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.050306 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:39.146967 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.146942 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nzwqh"] Jul 10 17:43:39.149206 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:39.149177 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91d859f_8d57_458c_8d15_2c124ece69c7.slice/crio-cba3f170b47dabb6ffa543ee8014f6dfd9d47c5cf6e9cbaacc6ec39bbf176d2c WatchSource:0}: Error finding container cba3f170b47dabb6ffa543ee8014f6dfd9d47c5cf6e9cbaacc6ec39bbf176d2c: Status 404 returned error can't find the container with id cba3f170b47dabb6ffa543ee8014f6dfd9d47c5cf6e9cbaacc6ec39bbf176d2c Jul 10 17:43:39.149836 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.149797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:39.212573 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.212548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:39.385248 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.385181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:39.425368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.425336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2f6c" event={"ID":"ba52d2bc-622c-430e-b437-aa91152f4524","Type":"ContainerStarted","Data":"92d3839fc5ea0b1e4c736a8a8f4f24e9b0006809bea4b5667719196fe4a2090d"} Jul 10 17:43:39.425519 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.425395 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2f6c" event={"ID":"ba52d2bc-622c-430e-b437-aa91152f4524","Type":"ContainerStarted","Data":"65c34cb30534b79e05578a14e213a364166ccde251c511f2f66d3e5ce40944d0"} Jul 10 17:43:39.426407 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.426380 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzwqh" event={"ID":"d91d859f-8d57-458c-8d15-2c124ece69c7","Type":"ContainerStarted","Data":"cba3f170b47dabb6ffa543ee8014f6dfd9d47c5cf6e9cbaacc6ec39bbf176d2c"} Jul 10 17:43:39.442608 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:39.442570 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2f6c" podStartSLOduration=2.377471558 podStartE2EDuration="3.442557669s" totalImagesPullingTime="994.808612ms" totalInitContainerRuntime="70.277499ms" isStatefulPod=true podCreationTimestamp="2026-07-10 17:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:39.441250576 +0000 UTC m=+286.502065911" watchObservedRunningTime="2026-07-10 17:43:39.442557669 +0000 UTC m=+286.503373003" Jul 10 17:43:40.591752 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:40.591716 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:40.699144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:40.699110 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:40.853019 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:40.852929 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:40.919695 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:40.919654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:40.982729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:40.982694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:41.435028 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:41.434985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzwqh" event={"ID":"d91d859f-8d57-458c-8d15-2c124ece69c7","Type":"ContainerStarted","Data":"41ce9b1aade3fc1904e4c478dc889c15a68b6393631933a8ee284b46bea8f13d"} Jul 10 17:43:41.435196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:41.435047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nzwqh" event={"ID":"d91d859f-8d57-458c-8d15-2c124ece69c7","Type":"ContainerStarted","Data":"030855fde2ac684c599af735e5927dd5dff1bd1f189f35f43f2f451a6c1526a2"} Jul 10 17:43:41.435196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:41.435101 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:42.331376 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.331312 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nzwqh" podStartSLOduration=252.466831342 podStartE2EDuration="4m14.331294038s" totalImagesPullingTime="1.864462696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:41.452487648 +0000 UTC m=+288.513302988" watchObservedRunningTime="2026-07-10 17:43:42.331294038 +0000 UTC m=+289.392109370" Jul 10 17:43:42.331840 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.331486 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:42.332194 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.332179 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:43:42.335514 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.335500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.337855 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.337824 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 17:43:42.337982 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.337833 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 17:43:42.337982 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.337901 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 17:43:42.337982 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.337938 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wc8hk\"" Jul 10 17:43:42.338129 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338005 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 17:43:42.338783 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338759 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4upkfgm02mpkt\"" Jul 10 17:43:42.338783 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338769 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 17:43:42.338935 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338788 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 17:43:42.338935 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338797 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 17:43:42.339064 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.338951 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 17:43:42.339243 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339223 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 17:43:42.339347 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339228 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 17:43:42.339347 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339278 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 17:43:42.339347 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339307 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 17:43:42.339492 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339275 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 17:43:42.339679 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.339663 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 17:43:42.342040 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.342021 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 17:43:42.351853 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.351830 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:43:42.430568 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430568 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430568 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430588 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430662 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rblg\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430818 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430879 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.430980 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.430981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.431354 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.431015 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532227 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532227 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rblg\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532549 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532571 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532636 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532630 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532722 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532774 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.532973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.532884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.533484 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.533466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.534064 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.534042 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.534684 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.534660 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.535276 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.535256 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.535590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.535685 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.535934 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.536130 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.536397 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536847 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.536557 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.536847 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.536599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.537394 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.537370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.538208 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.538182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.538703 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.538678 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.538791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.538736 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.539035 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.538984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.539145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.539047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.539145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.539064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.539316 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.539295 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.540076 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.540057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rblg\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg\") pod \"prometheus-k8s-0\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.645358 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.645281 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:42.777509 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:42.777476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:43:42.780406 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:43:42.780369 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b23a1a_3f6b_42f3_962f_66ccb621f157.slice/crio-4bf25b52a9636f9eca4608858f3a2f13ce304b53329fc9df80600cfa71399fea WatchSource:0}: Error finding container 4bf25b52a9636f9eca4608858f3a2f13ce304b53329fc9df80600cfa71399fea: Status 404 returned error can't find the container with id 4bf25b52a9636f9eca4608858f3a2f13ce304b53329fc9df80600cfa71399fea Jul 10 17:43:43.443527 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:43.443493 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"4bf25b52a9636f9eca4608858f3a2f13ce304b53329fc9df80600cfa71399fea"} Jul 10 17:43:44.448396 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:44.448363 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" exitCode=0 Jul 10 17:43:44.448748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:44.448412 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} Jul 10 17:43:44.452080 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:44.452059 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:45.727363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:45.727330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:45.941244 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:45.941204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:46.041756 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:46.041681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:47.115401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:47.115362 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:47.168688 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:47.168659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:47.428302 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:47.428258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wxtvv" podUID="62f47418-60ee-40ac-83f7-9b62d2a18531" Jul 10 17:43:47.570232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:47.570197 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:47.670291 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:47.670249 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:47.684539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:47.684470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:48.034573 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:48.034483 2582 patch_prober.go:28] interesting pod/image-registry-6b8dfdb869-tdt7n container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 17:43:48.034573 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:48.034551 2582 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" podUID="7096471e-80b4-4b31-8594-88e19c4951da" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:43:48.464729 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:48.464701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} Jul 10 17:43:48.573775 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:48.573239 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:49.391975 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.391948 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b8dfdb869-tdt7n" Jul 10 17:43:49.472098 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.472068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} Jul 10 17:43:49.486355 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.486318 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:49.582286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.582245 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:49.699565 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.699534 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:49.757370 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:49.757320 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:50.479146 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:50.479119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} Jul 10 17:43:50.479565 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:50.479176 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} Jul 10 17:43:50.479565 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:50.479203 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} Jul 10 17:43:50.479565 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:50.479229 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerStarted","Data":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} Jul 10 17:43:50.507076 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:50.507027 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.079094262 podStartE2EDuration="8.507006893s" totalImagesPullingTime="6.405807149s" totalInitContainerRuntime="22.105482ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:43:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:43:50.50588954 +0000 UTC m=+297.566704890" watchObservedRunningTime="2026-07-10 17:43:50.507006893 +0000 UTC m=+297.567822221" Jul 10 17:43:51.441196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:51.441172 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nzwqh" Jul 10 17:43:51.542465 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:51.542437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:51.644518 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:51.644484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:51.704903 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:51.704824 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:52.645698 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:52.645671 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:43:52.703309 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:52.703280 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6744d557b8-864vw" podUID="86ee37d7-3eae-488a-baec-557ab007695a" containerName="registry" containerID="cri-o://723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510" gracePeriod=30 Jul 10 17:43:52.931158 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:52.931138 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:53.024410 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024382 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024410 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024418 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024445 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5kj5\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024476 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024493 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024521 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.024626 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024563 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.025062 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.024632 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration\") pod \"86ee37d7-3eae-488a-baec-557ab007695a\" (UID: \"86ee37d7-3eae-488a-baec-557ab007695a\") " Jul 10 17:43:53.025543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.025179 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:43:53.025651 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.025558 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:43:53.027204 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.027176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:43:53.027204 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.027173 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:43:53.027365 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.027241 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:43:53.027444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.027427 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:43:53.027501 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.027444 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5" (OuterVolumeSpecName: "kube-api-access-l5kj5") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "kube-api-access-l5kj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:43:53.033333 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.033313 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "86ee37d7-3eae-488a-baec-557ab007695a" (UID: "86ee37d7-3eae-488a-baec-557ab007695a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:43:53.126123 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126101 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ee37d7-3eae-488a-baec-557ab007695a-ca-trust-extracted\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126123 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126123 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-registry-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126131 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5kj5\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-kube-api-access-l5kj5\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126142 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ee37d7-3eae-488a-baec-557ab007695a-bound-sa-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126150 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-registry-certificates\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126158 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-installation-pull-secrets\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126166 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ee37d7-3eae-488a-baec-557ab007695a-trusted-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.126241 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.126176 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ee37d7-3eae-488a-baec-557ab007695a-image-registry-private-configuration\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:43:53.313330 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.313264 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:43:53.314644 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.314623 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:43:53.320137 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.320118 2582 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 17:43:53.490202 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.490171 2582 generic.go:356] "Generic (PLEG): container finished" podID="86ee37d7-3eae-488a-baec-557ab007695a" containerID="723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510" exitCode=0 Jul 10 17:43:53.490373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.490214 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6744d557b8-864vw" Jul 10 17:43:53.490373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.490241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6744d557b8-864vw" event={"ID":"86ee37d7-3eae-488a-baec-557ab007695a","Type":"ContainerDied","Data":"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510"} Jul 10 17:43:53.490373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.490294 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6744d557b8-864vw" event={"ID":"86ee37d7-3eae-488a-baec-557ab007695a","Type":"ContainerDied","Data":"51bc2263daa502dfa049700d042307b237bb7622d746c751769dbb43b2b44e28"} Jul 10 17:43:53.490373 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.490319 2582 scope.go:117] "RemoveContainer" containerID="723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510" Jul 10 17:43:53.500047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.500025 2582 scope.go:117] "RemoveContainer" containerID="723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510" Jul 10 17:43:53.505461 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:43:53.500429 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510\": container with ID starting with 723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510 not found: ID does not exist" containerID="723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510" Jul 10 17:43:53.505461 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.502615 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510"} err="failed to get container status \"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510\": rpc error: code = NotFound desc = could not find container \"723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510\": container with ID starting with 723bcad39ccfbeca2c4d9a99722e6a8211ce0bb5ee87cb71489298b1aec1c510 not found: ID does not exist" Jul 10 17:43:53.510394 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.510369 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:43:53.514139 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:53.514119 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6744d557b8-864vw"] Jul 10 17:43:54.821173 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:54.821134 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:54.874213 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:54.874184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:55.210804 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:55.210770 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:55.264813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:55.264785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:55.431358 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:55.431323 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ee37d7-3eae-488a-baec-557ab007695a" path="/var/lib/kubelet/pods/86ee37d7-3eae-488a-baec-557ab007695a/volumes" Jul 10 17:43:58.427460 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:58.427430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:43:59.138577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:59.138543 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:43:59.190352 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:43:59.190321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:00.738837 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:00.738809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:00.794474 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:00.794447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:01.545798 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:01.545768 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:01.646863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:01.646832 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:05.988944 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:05.988913 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:06.089103 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:06.089064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:10.653576 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:10.653545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:10.752049 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:10.752017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:14.138368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:14.138339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:14.188759 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:14.188732 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:17.481708 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:17.481678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:23.588400 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:23.588369 2582 generic.go:356] "Generic (PLEG): container finished" podID="ba6b12af-bddd-4a83-b719-f11bb49d2eec" containerID="827b256225b18337843ed09d36175f89438509404edd73983a20b74891206f9f" exitCode=0 Jul 10 17:44:23.588807 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:23.588421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" event={"ID":"ba6b12af-bddd-4a83-b719-f11bb49d2eec","Type":"ContainerDied","Data":"827b256225b18337843ed09d36175f89438509404edd73983a20b74891206f9f"} Jul 10 17:44:23.588807 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:23.588694 2582 scope.go:117] "RemoveContainer" containerID="827b256225b18337843ed09d36175f89438509404edd73983a20b74891206f9f" Jul 10 17:44:24.593837 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:24.593805 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mzfgx" event={"ID":"ba6b12af-bddd-4a83-b719-f11bb49d2eec","Type":"ContainerStarted","Data":"855e0f244fa8ba94219270c52a512a91a21a97f84eff2ddc8c08a6fc43cc97b2"} Jul 10 17:44:29.145093 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:29.145053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:29.200745 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:29.200720 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:42.645793 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:42.645748 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:44:42.665920 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:42.665898 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:44:42.684685 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:42.684650 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:44:52.213545 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:52.213464 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:52.266850 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:52.266821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:59.113609 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:59.113579 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:44:59.214379 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:44:59.214351 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:00.720658 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.720627 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:00.721095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.720950 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="prometheus" containerID="cri-o://c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" gracePeriod=600 Jul 10 17:45:00.721095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.721012 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="thanos-sidecar" containerID="cri-o://52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" gracePeriod=600 Jul 10 17:45:00.721095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.721030 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="config-reloader" containerID="cri-o://9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" gracePeriod=600 Jul 10 17:45:00.721095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.721043 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-thanos" containerID="cri-o://461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" gracePeriod=600 Jul 10 17:45:00.721095 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.720963 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy" containerID="cri-o://f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" gracePeriod=600 Jul 10 17:45:00.721368 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.721021 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-web" containerID="cri-o://4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" gracePeriod=600 Jul 10 17:45:00.946906 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:00.946885 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.053687 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053598 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.053687 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053632 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.053687 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053657 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.053687 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053685 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053831 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053866 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rblg\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053891 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053934 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.053960 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054023 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054055 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054118 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054146 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054176 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054262 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054233 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054277 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.054634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054340 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy\") pod \"27b23a1a-3f6b-42f3-962f-66ccb621f157\" (UID: \"27b23a1a-3f6b-42f3-962f-66ccb621f157\") " Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054142 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.054839 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.055460 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.055794 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057325 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.056743 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:45:01.057737 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.057427 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.057737 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.057482 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out" (OuterVolumeSpecName: "config-out") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:45:01.057737 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.057716 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.058106 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058067 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:45:01.058106 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058084 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.058430 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058386 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.058430 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058405 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.058580 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058457 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg" (OuterVolumeSpecName: "kube-api-access-4rblg") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "kube-api-access-4rblg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:45:01.058580 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058523 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:45:01.058669 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058606 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.058766 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.058750 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.059160 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.059144 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.060074 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.060050 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config" (OuterVolumeSpecName: "config") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.068819 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.068800 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config" (OuterVolumeSpecName: "web-config") pod "27b23a1a-3f6b-42f3-962f-66ccb621f157" (UID: "27b23a1a-3f6b-42f3-962f-66ccb621f157"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:45:01.155290 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155263 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155292 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155307 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rblg\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-kube-api-access-4rblg\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155320 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155333 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155345 2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155362 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/27b23a1a-3f6b-42f3-962f-66ccb621f157-tls-assets\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155374 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-config-out\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155388 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155401 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155400 2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-config\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155413 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155428 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-db\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155441 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-web-config\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155453 2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-metrics-client-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155466 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-configmap-metrics-client-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155481 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155494 2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-kube-rbac-proxy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155508 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155520 2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/27b23a1a-3f6b-42f3-962f-66ccb621f157-secret-grpc-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.155758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.155536 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b23a1a-3f6b-42f3-962f-66ccb621f157-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723721 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723748 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723756 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723764 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723772 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723779 2582 generic.go:356] "Generic (PLEG): container finished" podID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" exitCode=0 Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} Jul 10 17:45:01.723810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723803 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723931 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723949 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.723953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.724039 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} Jul 10 17:45:01.724540 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.724063 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"27b23a1a-3f6b-42f3-962f-66ccb621f157","Type":"ContainerDied","Data":"4bf25b52a9636f9eca4608858f3a2f13ce304b53329fc9df80600cfa71399fea"} Jul 10 17:45:01.731825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.731809 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.743860 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.743839 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:01.744931 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.744916 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.747681 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.747663 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:01.751263 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.751249 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.757524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.757506 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.763883 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.763857 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.772541 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.772522 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.773171 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773151 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:01.773390 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773377 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="thanos-sidecar" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773392 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="thanos-sidecar" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773402 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="prometheus" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773408 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="prometheus" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773417 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-web" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773425 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-web" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773433 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773442 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773457 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-thanos" Jul 10 17:45:01.773459 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773462 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-thanos" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773473 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="init-config-reloader" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773478 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="init-config-reloader" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773484 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="config-reloader" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773489 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="config-reloader" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773497 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ee37d7-3eae-488a-baec-557ab007695a" containerName="registry" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773501 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ee37d7-3eae-488a-baec-557ab007695a" containerName="registry" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773544 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="prometheus" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773552 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="config-reloader" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773557 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773562 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-thanos" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773568 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="thanos-sidecar" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773605 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" containerName="kube-rbac-proxy-web" Jul 10 17:45:01.773730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.773618 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ee37d7-3eae-488a-baec-557ab007695a" containerName="registry" Jul 10 17:45:01.777272 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.777257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.779705 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779584 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 17:45:01.779832 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779807 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.779950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779909 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 17:45:01.779950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 17:45:01.779950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779929 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 17:45:01.780142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779968 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 17:45:01.780142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779985 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 17:45:01.780142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.779932 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 17:45:01.780348 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.780322 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.780438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780359 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.780438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780376 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 17:45:01.780438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780383 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.780438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780427 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4upkfgm02mpkt\"" Jul 10 17:45:01.780655 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780442 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wc8hk\"" Jul 10 17:45:01.780655 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780585 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 17:45:01.780744 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780655 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 17:45:01.780744 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780666 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 17:45:01.780744 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.780709 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.780890 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780740 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.780890 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780782 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.780890 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780877 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 17:45:01.781167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.780902 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 17:45:01.781167 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.781102 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.781167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781129 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.781167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781151 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.781436 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.781402 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.781543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781432 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.781543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781455 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.781832 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.781763 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.781832 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781794 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.781832 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.781815 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.782438 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.782414 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.782438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.782443 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.782692 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.782459 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.782813 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:45:01.782746 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.782813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.782772 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.782813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.782790 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.783077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783027 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.783077 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783057 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.783307 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783289 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.783384 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783310 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.783585 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783561 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.783673 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783588 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.783947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783890 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.783947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.783926 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.784255 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784227 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.784344 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784256 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.784563 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784540 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.784563 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784563 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.784708 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784656 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 17:45:01.784894 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784874 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.785023 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.784894 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.785298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785259 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.785298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785288 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.785620 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785597 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.785721 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785624 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.785888 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785865 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.785888 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.785890 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.786173 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786157 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.786252 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786174 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.786418 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786397 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.786474 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786421 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.786635 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786616 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 17:45:01.786721 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786661 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.786721 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786677 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.786907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786889 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.786981 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.786909 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.787198 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787181 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.787246 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787199 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.787396 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787373 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.787396 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787388 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.787588 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787570 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.787659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787590 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.787762 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787746 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.787798 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787763 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.787985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.787967 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.788098 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788030 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.788264 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788247 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.788324 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788265 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.788499 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788480 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.788499 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788498 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.788718 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788700 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.788764 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788720 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.788883 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788867 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.788928 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.788884 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.789116 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789099 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.789159 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789117 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.789317 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789301 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.789363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789317 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.789543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789525 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.789609 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789545 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.789759 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789733 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.789759 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789752 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.789947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789927 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.790042 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.789949 2582 scope.go:117] "RemoveContainer" containerID="461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479" Jul 10 17:45:01.790204 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790186 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479"} err="failed to get container status \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": rpc error: code = NotFound desc = could not find container \"461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479\": container with ID starting with 461e1c1ee5d58698055cd3ed2236145175ff8c9de83bcf1f2b8f551a9fd62479 not found: ID does not exist" Jul 10 17:45:01.790264 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790205 2582 scope.go:117] "RemoveContainer" containerID="f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994" Jul 10 17:45:01.790414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790396 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994"} err="failed to get container status \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": rpc error: code = NotFound desc = could not find container \"f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994\": container with ID starting with f2660c84eb0eae49d34c72398e0ca2242bb281813710a12f9152e03660ba9994 not found: ID does not exist" Jul 10 17:45:01.790472 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790416 2582 scope.go:117] "RemoveContainer" containerID="4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64" Jul 10 17:45:01.790662 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790645 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64"} err="failed to get container status \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": rpc error: code = NotFound desc = could not find container \"4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64\": container with ID starting with 4413d1145eabd93f7049d90eec23c26776e712a3ac4bf3a437d07b35917dda64 not found: ID does not exist" Jul 10 17:45:01.790713 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790663 2582 scope.go:117] "RemoveContainer" containerID="52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800" Jul 10 17:45:01.790857 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790841 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800"} err="failed to get container status \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": rpc error: code = NotFound desc = could not find container \"52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800\": container with ID starting with 52f00aea96e847ad6f50aea2439b9083ac5544df88bd395ef5a59f964cd2b800 not found: ID does not exist" Jul 10 17:45:01.790903 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.790857 2582 scope.go:117] "RemoveContainer" containerID="9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9" Jul 10 17:45:01.791049 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.791033 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9"} err="failed to get container status \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": rpc error: code = NotFound desc = could not find container \"9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9\": container with ID starting with 9bcff547fef7f3543dedd0b5ec7ca18d33150c01c1269dfe91d0541a122273d9 not found: ID does not exist" Jul 10 17:45:01.791103 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.791050 2582 scope.go:117] "RemoveContainer" containerID="c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e" Jul 10 17:45:01.791256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.791235 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e"} err="failed to get container status \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": rpc error: code = NotFound desc = could not find container \"c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e\": container with ID starting with c534ba789aed4fb9d1696c734745bb66bc969e51076ea5af990120d7f64c712e not found: ID does not exist" Jul 10 17:45:01.791300 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.791257 2582 scope.go:117] "RemoveContainer" containerID="31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2" Jul 10 17:45:01.791464 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.791448 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2"} err="failed to get container status \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": rpc error: code = NotFound desc = could not find container \"31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2\": container with ID starting with 31b5583174b41d230b08b229e9a0b0ad877125555e955d79d71e17dad6cacca2 not found: ID does not exist" Jul 10 17:45:01.798113 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.798088 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:01.862012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.861967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862143 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862165 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862191 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-web-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78fs\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-kube-api-access-w78fs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862375 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862471 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862531 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config-out\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.862730 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.862548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963339 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963477 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963477 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963477 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963408 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963654 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config-out\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963654 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963654 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963669 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.963791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963845 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-web-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78fs\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-kube-api-access-w78fs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963907 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963930 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.963965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.964012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.964043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964467 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.964320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.964792 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.964938 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.964877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.966818 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.966783 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.966956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.966936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config-out\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967173 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967340 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967312 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967529 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967584 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967644 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967611 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967800 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967783 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.967930 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.967905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.968476 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.968164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a03920cb-63ee-4314-8fc3-7ca20f0c8254-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.968476 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.968271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.968476 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.968317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-web-config\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.969754 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.969727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.969985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.969966 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.970287 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.970271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a03920cb-63ee-4314-8fc3-7ca20f0c8254-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.970585 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.970571 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:01.975830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:01.975786 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78fs\" (UniqueName: \"kubernetes.io/projected/a03920cb-63ee-4314-8fc3-7ca20f0c8254-kube-api-access-w78fs\") pod \"prometheus-k8s-0\" (UID: \"a03920cb-63ee-4314-8fc3-7ca20f0c8254\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:02.088164 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:02.088130 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:02.223796 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:02.223770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 17:45:02.227492 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:45:02.227430 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03920cb_63ee_4314_8fc3_7ca20f0c8254.slice/crio-3a156c6c4a6ec0b8642508aa7e35a63da01dd7bfa91e173d5cf0de0d72c3fbb7 WatchSource:0}: Error finding container 3a156c6c4a6ec0b8642508aa7e35a63da01dd7bfa91e173d5cf0de0d72c3fbb7: Status 404 returned error can't find the container with id 3a156c6c4a6ec0b8642508aa7e35a63da01dd7bfa91e173d5cf0de0d72c3fbb7 Jul 10 17:45:02.728374 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:02.728341 2582 generic.go:356] "Generic (PLEG): container finished" podID="a03920cb-63ee-4314-8fc3-7ca20f0c8254" containerID="e4cf30eafd9b9b72333827b53b53b0376656f05100c45df9f4ebfc26bdbaecab" exitCode=0 Jul 10 17:45:02.728825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:02.728431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerDied","Data":"e4cf30eafd9b9b72333827b53b53b0376656f05100c45df9f4ebfc26bdbaecab"} Jul 10 17:45:02.728825 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:02.728500 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"3a156c6c4a6ec0b8642508aa7e35a63da01dd7bfa91e173d5cf0de0d72c3fbb7"} Jul 10 17:45:03.435226 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.434478 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b23a1a-3f6b-42f3-962f-66ccb621f157" path="/var/lib/kubelet/pods/27b23a1a-3f6b-42f3-962f-66ccb621f157/volumes" Jul 10 17:45:03.736893 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.736816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"9e3218efdad707588b1760c112cada94e747ab947f400e39b3c38e7f69eb6e4f"} Jul 10 17:45:03.736893 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.736887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"8bac48972dfb8df1ed9c84e225cee970757170edd689c0772963714ca3945ce9"} Jul 10 17:45:03.737286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.736915 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"a22979db7bb1b6241c9e397f5ed7fdb2f49afd1ddcb5fcc71f00a3a8d3f73df7"} Jul 10 17:45:03.737286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.736948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"bcd3157ba3516618d6104cbb749128ff95cc4f00b3faab018a3bb69671a35fed"} Jul 10 17:45:03.737286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.736972 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"bfe6f888e3548d74942e67a848b6fd2189c06574ff95bbaefc20c7722d2005f1"} Jul 10 17:45:03.737286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.737022 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a03920cb-63ee-4314-8fc3-7ca20f0c8254","Type":"ContainerStarted","Data":"07d355f24189ad71024d1a88903528a0345e3637c1ea87edb4842a0e42301468"} Jul 10 17:45:03.763432 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:03.763378 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.745794956 podStartE2EDuration="2.763313766s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.51881ms" isStatefulPod=false podCreationTimestamp="2026-07-10 17:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:03.761810005 +0000 UTC m=+370.822625353" watchObservedRunningTime="2026-07-10 17:45:03.763313766 +0000 UTC m=+370.824129101" Jul 10 17:45:04.035142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:04.035069 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:04.087716 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:04.087687 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:07.088816 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.088788 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:45:07.208379 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.208350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:45:07.210635 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.210613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62f47418-60ee-40ac-83f7-9b62d2a18531-metrics-certs\") pod \"network-metrics-daemon-wxtvv\" (UID: \"62f47418-60ee-40ac-83f7-9b62d2a18531\") " pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:45:07.340382 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.340307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:07.430906 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.430886 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rxjk8\"" Jul 10 17:45:07.438813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.438800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wxtvv" Jul 10 17:45:07.585739 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.585706 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wxtvv"] Jul 10 17:45:07.588478 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:45:07.588447 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f47418_60ee_40ac_83f7_9b62d2a18531.slice/crio-aad35a059849a50024ca652705fcd33baaa4cb15a5bb13e0e4d3111bafa7bfe3 WatchSource:0}: Error finding container aad35a059849a50024ca652705fcd33baaa4cb15a5bb13e0e4d3111bafa7bfe3: Status 404 returned error can't find the container with id aad35a059849a50024ca652705fcd33baaa4cb15a5bb13e0e4d3111bafa7bfe3 Jul 10 17:45:07.590863 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.590850 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:45:07.751690 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:07.751657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wxtvv" event={"ID":"62f47418-60ee-40ac-83f7-9b62d2a18531","Type":"ContainerStarted","Data":"aad35a059849a50024ca652705fcd33baaa4cb15a5bb13e0e4d3111bafa7bfe3"} Jul 10 17:45:08.757307 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:08.757213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wxtvv" event={"ID":"62f47418-60ee-40ac-83f7-9b62d2a18531","Type":"ContainerStarted","Data":"fa123294d39d79fc64e25147a1482af07b80101707ea910bbdc8f2a745ef2369"} Jul 10 17:45:08.757307 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:08.757271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wxtvv" event={"ID":"62f47418-60ee-40ac-83f7-9b62d2a18531","Type":"ContainerStarted","Data":"ba567054ae8ffdb67ad960c28bc55fe322c6edf0ca1724d7e3a7f3f1bf8f01db"} Jul 10 17:45:08.771907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:08.771858 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wxtvv" podStartSLOduration=373.909997573 podStartE2EDuration="6m14.771844537s" totalImagesPullingTime="861.846964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:45:08.771029774 +0000 UTC m=+375.831845114" watchObservedRunningTime="2026-07-10 17:45:08.771844537 +0000 UTC m=+375.832659872" Jul 10 17:45:08.863907 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:08.863872 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:08.940901 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:08.940869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:14.160457 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:14.160415 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:14.254678 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:14.254646 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:19.828981 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:19.828952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:19.879943 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:19.879915 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:19.969964 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:19.969934 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:20.071973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:20.071942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:20.564726 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:20.564694 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:20.615569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:20.615542 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:30.913316 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:30.913283 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:31.018805 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:31.018772 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:31.077125 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:31.077094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:44.836754 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:44.836717 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:45:44.888121 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:45:44.888094 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:02.088288 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:02.088251 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:46:02.103846 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:02.103821 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:46:02.967290 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:02.967250 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 17:46:16.045651 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:16.045573 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:16.144481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:16.144447 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:42.689178 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:42.689138 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:42.791947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:42.791914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.514236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:46.514197 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:46.615213 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:46.615181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:47.872611 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:47.872578 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:47.976553 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:47.976523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:48.349611 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:48.349574 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:48.452634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:48.452596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:50.734148 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:50.734116 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:50.939203 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:50.939173 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:52.689820 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:52.689786 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:52.746132 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:52.746102 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:56.546569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:56.546537 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:56.644316 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:56.644287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:58.331464 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:58.331434 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:59.644824 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:59.644792 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:46:59.742781 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:46:59.742752 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:26.966374 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:26.966335 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:27.067159 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:27.067128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:28.868724 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:28.868689 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:31.809740 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:31.809695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:33.168924 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:33.168891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:33.249321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:33.249297 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:42.128537 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:42.128504 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:42.235046 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:42.235018 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:42.787884 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:42.787846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:42.986039 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:42.986010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.092280 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.092203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.111589 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.111564 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b"] Jul 10 17:47:43.114576 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.114558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.116887 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.116863 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 17:47:43.117167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.117137 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7wd7s\"" Jul 10 17:47:43.117320 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.117222 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 17:47:43.117320 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.117298 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 17:47:43.117438 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.117412 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 17:47:43.132133 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.132115 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b"] Jul 10 17:47:43.153140 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.153119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc297\" (UniqueName: \"kubernetes.io/projected/5f0dca97-484c-4a63-81ee-2959b7867566-kube-api-access-lc297\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.153236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.153159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.153236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.153212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.153305 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.153262 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.153337 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.153300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.254425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc297\" (UniqueName: \"kubernetes.io/projected/5f0dca97-484c-4a63-81ee-2959b7867566-kube-api-access-lc297\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.254615 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254437 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.254615 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254561 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.254615 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.254804 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.255013 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.254969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-manifests\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.256971 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.256940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f0dca97-484c-4a63-81ee-2959b7867566-tmp\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.257096 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.257057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.257143 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.257114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f0dca97-484c-4a63-81ee-2959b7867566-webhook-cert\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.276543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.276515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc297\" (UniqueName: \"kubernetes.io/projected/5f0dca97-484c-4a63-81ee-2959b7867566-kube-api-access-lc297\") pod \"opendatahub-operator-controller-manager-dfdbc665d-xfx4b\" (UID: \"5f0dca97-484c-4a63-81ee-2959b7867566\") " pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.345561 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.345469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.424302 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.424281 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:43.464676 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.464644 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.547978 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.547944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.559388 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.559360 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b"] Jul 10 17:47:43.562763 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:47:43.562736 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0dca97_484c_4a63_81ee_2959b7867566.slice/crio-c04f5b15cba9b72dc688e290f35d11e6adaf8b02e7580c3d019e228062827648 WatchSource:0}: Error finding container c04f5b15cba9b72dc688e290f35d11e6adaf8b02e7580c3d019e228062827648: Status 404 returned error can't find the container with id c04f5b15cba9b72dc688e290f35d11e6adaf8b02e7580c3d019e228062827648 Jul 10 17:47:43.653167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.653145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.756003 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.755976 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:43.854548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:43.854524 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:44.296748 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:44.296653 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" event={"ID":"5f0dca97-484c-4a63-81ee-2959b7867566","Type":"ContainerStarted","Data":"c04f5b15cba9b72dc688e290f35d11e6adaf8b02e7580c3d019e228062827648"} Jul 10 17:47:47.310272 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:47.310243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" event={"ID":"5f0dca97-484c-4a63-81ee-2959b7867566","Type":"ContainerStarted","Data":"343921c450d072d484ad845c348533810b820c654431dcab919caf86cc8fb301"} Jul 10 17:47:48.314886 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:48.314853 2582 generic.go:356] "Generic (PLEG): container finished" podID="5f0dca97-484c-4a63-81ee-2959b7867566" containerID="343921c450d072d484ad845c348533810b820c654431dcab919caf86cc8fb301" exitCode=0 Jul 10 17:47:48.315322 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:48.314892 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" event={"ID":"5f0dca97-484c-4a63-81ee-2959b7867566","Type":"ContainerDied","Data":"343921c450d072d484ad845c348533810b820c654431dcab919caf86cc8fb301"} Jul 10 17:47:49.319950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:49.319911 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" event={"ID":"5f0dca97-484c-4a63-81ee-2959b7867566","Type":"ContainerStarted","Data":"12924196d8dbf93486b7a95b83d5340db91031519adf2f04c4f1f27005140cc5"} Jul 10 17:47:49.320353 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:49.320125 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:47:49.340900 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:49.340851 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" podStartSLOduration=1.975934898 podStartE2EDuration="6.340834641s" totalImagesPullingTime="3.322424727s" totalInitContainerRuntime="1.042475016s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:47:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:47:49.340498069 +0000 UTC m=+536.401313403" watchObservedRunningTime="2026-07-10 17:47:49.340834641 +0000 UTC m=+536.401649985" Jul 10 17:47:54.296076 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:54.296029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:47:54.408574 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:47:54.408546 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:00.326504 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:00.326474 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dfdbc665d-xfx4b" Jul 10 17:48:01.982959 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:01.982924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:02.262586 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.262509 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:02.515956 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.515875 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:02.547564 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.547538 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-prll8"] Jul 10 17:48:02.556946 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.556923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.559749 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.559728 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 17:48:02.559852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.559794 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 17:48:02.560586 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.560572 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 17:48:02.563523 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.563372 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 17:48:02.570525 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.570501 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-prll8"] Jul 10 17:48:02.607864 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.607838 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d008591-1495-44c0-a02e-81631131495a-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.607986 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.607882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgpj\" (UniqueName: \"kubernetes.io/projected/3d008591-1495-44c0-a02e-81631131495a-kube-api-access-swgpj\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.607986 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.607913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d008591-1495-44c0-a02e-81631131495a-tmp\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.708978 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.708945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d008591-1495-44c0-a02e-81631131495a-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.709168 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.709018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgpj\" (UniqueName: \"kubernetes.io/projected/3d008591-1495-44c0-a02e-81631131495a-kube-api-access-swgpj\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.709168 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.709051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d008591-1495-44c0-a02e-81631131495a-tmp\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.711393 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.711371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d008591-1495-44c0-a02e-81631131495a-tmp\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.711511 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.711495 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d008591-1495-44c0-a02e-81631131495a-tls-certs\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.723278 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.723255 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgpj\" (UniqueName: \"kubernetes.io/projected/3d008591-1495-44c0-a02e-81631131495a-kube-api-access-swgpj\") pod \"kube-auth-proxy-5b544bffd9-prll8\" (UID: \"3d008591-1495-44c0-a02e-81631131495a\") " pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.762569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.762540 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:02.807936 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.807867 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8j6s9"] Jul 10 17:48:02.810046 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.810030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:02.812531 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.812507 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cr6wr\"" Jul 10 17:48:02.812638 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.812590 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 17:48:02.823349 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.823327 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8j6s9"] Jul 10 17:48:02.865911 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.865885 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" Jul 10 17:48:02.910412 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.910378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:02.910599 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.910487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkk7\" (UniqueName: \"kubernetes.io/projected/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-kube-api-access-4jkk7\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:02.942561 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:02.942531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.011046 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.011013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkk7\" (UniqueName: \"kubernetes.io/projected/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-kube-api-access-4jkk7\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.011432 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.011108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.011432 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:48:03.011214 2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 17:48:03.011432 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:48:03.011278 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert podName:26a44ad2-1fed-4d9e-a016-4890b2fb2ea0 nodeName:}" failed. No retries permitted until 2026-07-10 17:48:03.511257145 +0000 UTC m=+550.572072472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert") pod "odh-model-controller-7998c67dc4-8j6s9" (UID: "26a44ad2-1fed-4d9e-a016-4890b2fb2ea0") : secret "odh-model-controller-webhook-cert" not found Jul 10 17:48:03.052784 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.052760 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkk7\" (UniqueName: \"kubernetes.io/projected/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-kube-api-access-4jkk7\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.064434 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.064408 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b544bffd9-prll8"] Jul 10 17:48:03.066590 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:03.066563 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d008591_1495_44c0_a02e_81631131495a.slice/crio-3963292a263025cc66f30c8798b0121379724b4e52a9b7f88dce3c9925114294 WatchSource:0}: Error finding container 3963292a263025cc66f30c8798b0121379724b4e52a9b7f88dce3c9925114294: Status 404 returned error can't find the container with id 3963292a263025cc66f30c8798b0121379724b4e52a9b7f88dce3c9925114294 Jul 10 17:48:03.158666 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.158634 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.358053 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.357946 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:03.372778 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.372751 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" event={"ID":"3d008591-1495-44c0-a02e-81631131495a","Type":"ContainerStarted","Data":"3963292a263025cc66f30c8798b0121379724b4e52a9b7f88dce3c9925114294"} Jul 10 17:48:03.515871 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.515830 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.518392 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.518370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26a44ad2-1fed-4d9e-a016-4890b2fb2ea0-cert\") pod \"odh-model-controller-7998c67dc4-8j6s9\" (UID: \"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0\") " pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.719700 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.719666 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:03.869618 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:03.869356 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8j6s9"] Jul 10 17:48:03.872167 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:03.872133 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a44ad2_1fed_4d9e_a016_4890b2fb2ea0.slice/crio-114ed0452d8f65efaa02172fc193b79e81833a78d0f5f797a77dccfa9ce66722 WatchSource:0}: Error finding container 114ed0452d8f65efaa02172fc193b79e81833a78d0f5f797a77dccfa9ce66722: Status 404 returned error can't find the container with id 114ed0452d8f65efaa02172fc193b79e81833a78d0f5f797a77dccfa9ce66722 Jul 10 17:48:04.019311 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.019230 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:04.066257 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.066229 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cj854"] Jul 10 17:48:04.073834 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.073801 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.076830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.076813 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-scjml\"" Jul 10 17:48:04.077339 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.077325 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 17:48:04.094242 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.094221 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cj854"] Jul 10 17:48:04.121624 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.121594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684b129-1905-410d-a262-822ef62bcd0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.121771 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.121632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjgn\" (UniqueName: \"kubernetes.io/projected/a684b129-1905-410d-a262-822ef62bcd0d-kube-api-access-9mjgn\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.219305 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.219272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:04.222385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.222366 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684b129-1905-410d-a262-822ef62bcd0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.222445 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.222397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjgn\" (UniqueName: \"kubernetes.io/projected/a684b129-1905-410d-a262-822ef62bcd0d-kube-api-access-9mjgn\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.225145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.225125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a684b129-1905-410d-a262-822ef62bcd0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.248607 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.248576 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjgn\" (UniqueName: \"kubernetes.io/projected/a684b129-1905-410d-a262-822ef62bcd0d-kube-api-access-9mjgn\") pod \"model-serving-api-54d89bd5fc-cj854\" (UID: \"a684b129-1905-410d-a262-822ef62bcd0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.378830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.378662 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" event={"ID":"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0","Type":"ContainerStarted","Data":"114ed0452d8f65efaa02172fc193b79e81833a78d0f5f797a77dccfa9ce66722"} Jul 10 17:48:04.384520 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.384496 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:04.515949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.515923 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cj854"] Jul 10 17:48:04.518293 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:04.518271 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda684b129_1905_410d_a262_822ef62bcd0d.slice/crio-51fbd5eddbea789e54f5733e3a4a5459a99153ab03b13c64b69fc6139f02806d WatchSource:0}: Error finding container 51fbd5eddbea789e54f5733e3a4a5459a99153ab03b13c64b69fc6139f02806d: Status 404 returned error can't find the container with id 51fbd5eddbea789e54f5733e3a4a5459a99153ab03b13c64b69fc6139f02806d Jul 10 17:48:04.581299 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.581266 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:04.583702 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.583679 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.586582 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586283 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cttkq\"" Jul 10 17:48:04.586582 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586310 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 17:48:04.586582 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586358 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 17:48:04.586582 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586286 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 17:48:04.586582 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 17:48:04.586876 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.586566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:04.598890 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.598742 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:04.626460 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.626791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65sdt\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.626791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.626791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.626791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626744 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.626791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.627107 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.626889 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.659962 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.659883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727573 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727610 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727645 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727683 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65sdt\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.728597 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.727743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.729409 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.729360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.732591 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.732550 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.732904 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.732882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.733550 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.733478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.735710 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.735669 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.736830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.736806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.738958 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.738933 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65sdt\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt\") pod \"istiod-openshift-gateway-7574459656-428l5\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:04.897036 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:04.896743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:05.069142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:05.069072 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:05.072980 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:05.072941 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1378e845_614d_4e06_8eb1_5f0feda78323.slice/crio-24e5da71d377a081df66d944f053aacfcae06996b2f745b147c092ce932b82d2 WatchSource:0}: Error finding container 24e5da71d377a081df66d944f053aacfcae06996b2f745b147c092ce932b82d2: Status 404 returned error can't find the container with id 24e5da71d377a081df66d944f053aacfcae06996b2f745b147c092ce932b82d2 Jul 10 17:48:05.384679 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:05.384583 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" event={"ID":"1378e845-614d-4e06-8eb1-5f0feda78323","Type":"ContainerStarted","Data":"24e5da71d377a081df66d944f053aacfcae06996b2f745b147c092ce932b82d2"} Jul 10 17:48:05.385852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:05.385826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" event={"ID":"a684b129-1905-410d-a262-822ef62bcd0d","Type":"ContainerStarted","Data":"51fbd5eddbea789e54f5733e3a4a5459a99153ab03b13c64b69fc6139f02806d"} Jul 10 17:48:05.583140 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:05.583080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:08.891809 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:08.891774 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:48:08.892113 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:08.891839 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:48:09.405082 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.405041 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" event={"ID":"a684b129-1905-410d-a262-822ef62bcd0d","Type":"ContainerStarted","Data":"b09f2b6c83813f8eee1842d64a39041a6b14799972b74c329ef0ff1154beb165"} Jul 10 17:48:09.405254 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.405209 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:09.407007 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.406967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" event={"ID":"3d008591-1495-44c0-a02e-81631131495a","Type":"ContainerStarted","Data":"6ae1baba13dfb77d7ad7125978cab28d335a7eb6b8193710300e97f6fa6f1a31"} Jul 10 17:48:09.408955 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.408893 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" event={"ID":"26a44ad2-1fed-4d9e-a016-4890b2fb2ea0","Type":"ContainerStarted","Data":"a947f73c048f39aea3bad765569e69ec2c53109e1a58727851c38a636da62aec"} Jul 10 17:48:09.409108 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.408976 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:09.410647 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.410622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" event={"ID":"1378e845-614d-4e06-8eb1-5f0feda78323","Type":"ContainerStarted","Data":"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099"} Jul 10 17:48:09.410822 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.410808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:09.412515 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.412491 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-428l5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 17:48:09.412611 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.412543 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 17:48:09.427256 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.427213 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" podStartSLOduration=1.126315217 podStartE2EDuration="5.427200451s" totalImagesPullingTime="4.300885234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:09.42578521 +0000 UTC m=+556.486600546" watchObservedRunningTime="2026-07-10 17:48:09.427200451 +0000 UTC m=+556.488015768" Jul 10 17:48:09.456647 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.456586 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" podStartSLOduration=1.640494666 podStartE2EDuration="5.45656553s" totalImagesPullingTime="3.816070864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:09.45437493 +0000 UTC m=+556.515190249" watchObservedRunningTime="2026-07-10 17:48:09.45656553 +0000 UTC m=+556.517380869" Jul 10 17:48:09.475240 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.475167 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" podStartSLOduration=2.5271381550000003 podStartE2EDuration="7.475149786s" totalImagesPullingTime="4.948011631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:09.473849885 +0000 UTC m=+556.534665223" watchObservedRunningTime="2026-07-10 17:48:09.475149786 +0000 UTC m=+556.535965123" Jul 10 17:48:09.495133 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.495063 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b544bffd9-prll8" podStartSLOduration=1.742180557 podStartE2EDuration="7.495040872s" totalImagesPullingTime="5.752860315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:09.492780999 +0000 UTC m=+556.553596346" watchObservedRunningTime="2026-07-10 17:48:09.495040872 +0000 UTC m=+556.555856211" Jul 10 17:48:09.540893 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.540849 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:09.643698 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:09.643666 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:10.301892 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.301832 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt"] Jul 10 17:48:10.305121 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.305099 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.308662 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.308644 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t9qrs\"" Jul 10 17:48:10.339758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.339735 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt"] Jul 10 17:48:10.366755 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.366725 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:10.378365 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378338 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378467 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378467 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378467 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblqj\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-kube-api-access-wblqj\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378467 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378469 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5268379e-aff0-441a-b984-b806c13d2b65-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.378659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.378610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5268379e-aff0-441a-b984-b806c13d2b65-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.415468 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.415448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:10.479481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479481 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479736 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479803 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblqj\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-kube-api-access-wblqj\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479803 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479897 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.479897 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480073 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479963 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480073 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.479971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5268379e-aff0-441a-b984-b806c13d2b65-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480073 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.480043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.480125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5268379e-aff0-441a-b984-b806c13d2b65-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.480141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480236 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.480160 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.480844 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.480827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5268379e-aff0-441a-b984-b806c13d2b65-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.483446 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.483388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5268379e-aff0-441a-b984-b806c13d2b65-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.483446 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.483413 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5268379e-aff0-441a-b984-b806c13d2b65-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.491339 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.491316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblqj\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-kube-api-access-wblqj\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.496156 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.496132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5268379e-aff0-441a-b984-b806c13d2b65-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678pqxqt\" (UID: \"5268379e-aff0-441a-b984-b806c13d2b65\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.606646 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.606564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:10.615101 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.615074 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:10.756943 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:10.756921 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt"] Jul 10 17:48:10.759251 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:10.759229 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5268379e_aff0_441a_b984_b806c13d2b65.slice/crio-1f8ef0cef5b0bebc3c8491d5ae70a1eade70855cdd009ac2c433c32b1346c9b5 WatchSource:0}: Error finding container 1f8ef0cef5b0bebc3c8491d5ae70a1eade70855cdd009ac2c433c32b1346c9b5: Status 404 returned error can't find the container with id 1f8ef0cef5b0bebc3c8491d5ae70a1eade70855cdd009ac2c433c32b1346c9b5 Jul 10 17:48:11.152272 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:11.152241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:11.420697 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:11.420617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" event={"ID":"5268379e-aff0-441a-b984-b806c13d2b65","Type":"ContainerStarted","Data":"1f8ef0cef5b0bebc3c8491d5ae70a1eade70855cdd009ac2c433c32b1346c9b5"} Jul 10 17:48:13.092798 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.092767 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.181173 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.181128 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:48:13.181298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.181220 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:48:13.181298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.181263 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:48:13.202731 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.202706 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb"] Jul 10 17:48:13.207094 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.207072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.207604 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.207586 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s"] Jul 10 17:48:13.209659 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.209644 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xvjl9\"" Jul 10 17:48:13.209868 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.209854 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 17:48:13.210535 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.210521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.215949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.215931 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-z7wb4\"" Jul 10 17:48:13.217761 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.217742 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 17:48:13.230974 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.230951 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb"] Jul 10 17:48:13.235442 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.235420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s"] Jul 10 17:48:13.256926 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.256900 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.305657 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.305627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/986dc769-d84c-4d51-8a15-a7702bd0afc4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.305852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.305672 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zm7\" (UniqueName: \"kubernetes.io/projected/986dc769-d84c-4d51-8a15-a7702bd0afc4-kube-api-access-p9zm7\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.305852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.305784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.305852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.305823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbx8\" (UniqueName: \"kubernetes.io/projected/e733e6bc-ad67-489e-98fa-7821e493fa92-kube-api-access-wkbx8\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.406744 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.406716 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.406972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.406763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbx8\" (UniqueName: \"kubernetes.io/projected/e733e6bc-ad67-489e-98fa-7821e493fa92-kube-api-access-wkbx8\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.406972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.406810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/986dc769-d84c-4d51-8a15-a7702bd0afc4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.406972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.406854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zm7\" (UniqueName: \"kubernetes.io/projected/986dc769-d84c-4d51-8a15-a7702bd0afc4-kube-api-access-p9zm7\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.406972 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:48:13.406894 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 17:48:13.406972 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:48:13.406971 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert podName:e733e6bc-ad67-489e-98fa-7821e493fa92 nodeName:}" failed. No retries permitted until 2026-07-10 17:48:13.90694826 +0000 UTC m=+560.967763588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert") pod "kserve-controller-manager-d6f4b65b8-sgwmb" (UID: "e733e6bc-ad67-489e-98fa-7821e493fa92") : secret "kserve-webhook-server-cert" not found Jul 10 17:48:13.409493 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.409463 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/986dc769-d84c-4d51-8a15-a7702bd0afc4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.421061 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.421037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zm7\" (UniqueName: \"kubernetes.io/projected/986dc769-d84c-4d51-8a15-a7702bd0afc4-kube-api-access-p9zm7\") pod \"llmisvc-controller-manager-777c6bd7d5-czk4s\" (UID: \"986dc769-d84c-4d51-8a15-a7702bd0afc4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.421061 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.421053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbx8\" (UniqueName: \"kubernetes.io/projected/e733e6bc-ad67-489e-98fa-7821e493fa92-kube-api-access-wkbx8\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.422167 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.422145 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.433014 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.432966 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" event={"ID":"5268379e-aff0-441a-b984-b806c13d2b65","Type":"ContainerStarted","Data":"291189851dba2417907bcb4a3072644dc23f2dcc6ce3bd9066fad120d361d620"} Jul 10 17:48:13.489274 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.489225 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" podStartSLOduration=1.069675201 podStartE2EDuration="3.489209855s" totalImagesPullingTime="2.419534654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:13.488955973 +0000 UTC m=+560.549771312" watchObservedRunningTime="2026-07-10 17:48:13.489209855 +0000 UTC m=+560.550025189" Jul 10 17:48:13.541026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.540975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:13.557894 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.557871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.622017 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.620140 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:13.740130 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.740052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s"] Jul 10 17:48:13.749036 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:13.749003 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod986dc769_d84c_4d51_8a15_a7702bd0afc4.slice/crio-a0270d672ed43b7c885b3b37c93426e3a1ae76e92086375173e7c1e3112e1273 WatchSource:0}: Error finding container a0270d672ed43b7c885b3b37c93426e3a1ae76e92086375173e7c1e3112e1273: Status 404 returned error can't find the container with id a0270d672ed43b7c885b3b37c93426e3a1ae76e92086375173e7c1e3112e1273 Jul 10 17:48:13.755875 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.755853 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.909976 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.909941 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:48:13.913108 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.913084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:13.915539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:13.915512 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e733e6bc-ad67-489e-98fa-7821e493fa92-cert\") pod \"kserve-controller-manager-d6f4b65b8-sgwmb\" (UID: \"e733e6bc-ad67-489e-98fa-7821e493fa92\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:14.125972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:14.125897 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:14.253949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:14.253805 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb"] Jul 10 17:48:14.256094 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:48:14.256066 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode733e6bc_ad67_489e_98fa_7821e493fa92.slice/crio-4952c37a412a031a9a574aeb9da104ddea6c831659e1a1564c3b8246878bc513 WatchSource:0}: Error finding container 4952c37a412a031a9a574aeb9da104ddea6c831659e1a1564c3b8246878bc513: Status 404 returned error can't find the container with id 4952c37a412a031a9a574aeb9da104ddea6c831659e1a1564c3b8246878bc513 Jul 10 17:48:14.437793 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:14.437762 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" event={"ID":"e733e6bc-ad67-489e-98fa-7821e493fa92","Type":"ContainerStarted","Data":"4952c37a412a031a9a574aeb9da104ddea6c831659e1a1564c3b8246878bc513"} Jul 10 17:48:14.438870 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:14.438837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" event={"ID":"986dc769-d84c-4d51-8a15-a7702bd0afc4","Type":"ContainerStarted","Data":"a0270d672ed43b7c885b3b37c93426e3a1ae76e92086375173e7c1e3112e1273"} Jul 10 17:48:14.620293 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:14.620267 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:15.443580 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:15.443552 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:15.444653 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:15.444630 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678pqxqt" Jul 10 17:48:15.933715 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:15.933681 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:15.933904 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:15.933884 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" containerName="discovery" containerID="cri-o://fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099" gracePeriod=30 Jul 10 17:48:16.193085 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.193036 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:16.336385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336357 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336455 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336507 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65sdt\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336734 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336567 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336734 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336601 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.336734 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.336644 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap\") pod \"1378e845-614d-4e06-8eb1-5f0feda78323\" (UID: \"1378e845-614d-4e06-8eb1-5f0feda78323\") " Jul 10 17:48:16.337319 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.337265 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:48:16.339915 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.339868 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs" (OuterVolumeSpecName: "local-certs") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:48:16.339915 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.339888 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token" (OuterVolumeSpecName: "istio-token") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:48:16.340131 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.339912 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:48:16.341234 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.341138 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:48:16.341425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.341401 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt" (OuterVolumeSpecName: "kube-api-access-65sdt") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "kube-api-access-65sdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:48:16.341886 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.341856 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts" (OuterVolumeSpecName: "cacerts") pod "1378e845-614d-4e06-8eb1-5f0feda78323" (UID: "1378e845-614d-4e06-8eb1-5f0feda78323"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:48:16.437813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437780 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-dns-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.437813 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437812 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-istio-kubeconfig\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.438018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437826 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1378e845-614d-4e06-8eb1-5f0feda78323-cacerts\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.438018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437840 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65sdt\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-kube-api-access-65sdt\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.438018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437857 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1378e845-614d-4e06-8eb1-5f0feda78323-local-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.438018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437869 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1378e845-614d-4e06-8eb1-5f0feda78323-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.438018 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.437881 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1378e845-614d-4e06-8eb1-5f0feda78323-istio-csr-ca-configmap\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:48:16.448961 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.448902 2582 generic.go:356] "Generic (PLEG): container finished" podID="1378e845-614d-4e06-8eb1-5f0feda78323" containerID="fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099" exitCode=0 Jul 10 17:48:16.448961 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.448954 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" Jul 10 17:48:16.449429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.449016 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" event={"ID":"1378e845-614d-4e06-8eb1-5f0feda78323","Type":"ContainerDied","Data":"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099"} Jul 10 17:48:16.449429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.449072 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-428l5" event={"ID":"1378e845-614d-4e06-8eb1-5f0feda78323","Type":"ContainerDied","Data":"24e5da71d377a081df66d944f053aacfcae06996b2f745b147c092ce932b82d2"} Jul 10 17:48:16.449429 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.449100 2582 scope.go:117] "RemoveContainer" containerID="fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099" Jul 10 17:48:16.474629 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.474602 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:16.479124 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.479096 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-428l5"] Jul 10 17:48:16.957712 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.957605 2582 scope.go:117] "RemoveContainer" containerID="fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099" Jul 10 17:48:16.957978 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:48:16.957956 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099\": container with ID starting with fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099 not found: ID does not exist" containerID="fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099" Jul 10 17:48:16.958035 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:16.958004 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099"} err="failed to get container status \"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099\": rpc error: code = NotFound desc = could not find container \"fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099\": container with ID starting with fe9b9e775074b487363633b13672ca67ad97cb3eab8227db112e26f8ba405099 not found: ID does not exist" Jul 10 17:48:17.434549 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:17.434508 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" path="/var/lib/kubelet/pods/1378e845-614d-4e06-8eb1-5f0feda78323/volumes" Jul 10 17:48:18.459777 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.459745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" event={"ID":"e733e6bc-ad67-489e-98fa-7821e493fa92","Type":"ContainerStarted","Data":"b72cc980bb87f8e8be71319004a39a2beb20de6ccf3d30fc5318bd4daec5936d"} Jul 10 17:48:18.460247 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.459860 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:18.461305 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.461278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" event={"ID":"986dc769-d84c-4d51-8a15-a7702bd0afc4","Type":"ContainerStarted","Data":"de4e0cbbb153f3e8db9bc1cede2295ac3650ea629ade3fbddf67dfaf5eca48c3"} Jul 10 17:48:18.461386 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.461373 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:18.476250 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.476211 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" podStartSLOduration=2.1772930759999998 podStartE2EDuration="5.47619823s" totalImagesPullingTime="3.298905154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:18.475256647 +0000 UTC m=+565.536071990" watchObservedRunningTime="2026-07-10 17:48:18.47619823 +0000 UTC m=+565.537013564" Jul 10 17:48:18.495414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:18.495374 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" podStartSLOduration=1.686420827 podStartE2EDuration="5.495359471s" totalImagesPullingTime="3.808938644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:48:18.494885405 +0000 UTC m=+565.555700741" watchObservedRunningTime="2026-07-10 17:48:18.495359471 +0000 UTC m=+565.556174805" Jul 10 17:48:20.417226 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:20.417197 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8j6s9" Jul 10 17:48:20.419096 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:20.419078 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-cj854" Jul 10 17:48:49.470524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:49.470496 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-czk4s" Jul 10 17:48:49.473908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:49.473883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-sgwmb" Jul 10 17:48:53.339431 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:53.339403 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:48:53.339852 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:48:53.339694 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:49:00.232835 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:00.232797 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:01.512532 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:01.512502 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:01.981292 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:01.981242 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:03.458225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.458190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:03.568793 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.568758 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:03.621736 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.621710 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:03.622144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.622128 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" containerName="discovery" Jul 10 17:49:03.622197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.622146 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" containerName="discovery" Jul 10 17:49:03.622232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.622197 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1378e845-614d-4e06-8eb1-5f0feda78323" containerName="discovery" Jul 10 17:49:03.624161 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.624148 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:03.627876 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.627713 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 17:49:03.627876 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.627723 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-h7lfb\"" Jul 10 17:49:03.627876 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.627759 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 17:49:03.632848 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.632826 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:03.673918 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.673891 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:03.718890 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.718830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjpd\" (UniqueName: \"kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd\") pod \"kuadrant-operator-catalog-2kzmr\" (UID: \"65c3a02e-a646-428f-b7fb-544c4860f30c\") " pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:03.820149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.820115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjpd\" (UniqueName: \"kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd\") pod \"kuadrant-operator-catalog-2kzmr\" (UID: \"65c3a02e-a646-428f-b7fb-544c4860f30c\") " pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:03.828076 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.828057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjpd\" (UniqueName: \"kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd\") pod \"kuadrant-operator-catalog-2kzmr\" (UID: \"65c3a02e-a646-428f-b7fb-544c4860f30c\") " pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:03.935539 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.935511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:03.955433 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:03.955410 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:04.058971 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.058945 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:04.063320 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:04.063223 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c3a02e_a646_428f_b7fb_544c4860f30c.slice/crio-a079d71b0a3f386a979226bb0a1992ccb0a5165a994b4e3abba043621e931623 WatchSource:0}: Error finding container a079d71b0a3f386a979226bb0a1992ccb0a5165a994b4e3abba043621e931623: Status 404 returned error can't find the container with id a079d71b0a3f386a979226bb0a1992ccb0a5165a994b4e3abba043621e931623 Jul 10 17:49:04.165640 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.165607 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t4b9"] Jul 10 17:49:04.168550 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.168531 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:04.176061 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.176040 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t4b9"] Jul 10 17:49:04.224456 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.224434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728d7\" (UniqueName: \"kubernetes.io/projected/fd2246d1-f0b9-4817-a935-5dbe2f9b3577-kube-api-access-728d7\") pod \"kuadrant-operator-catalog-7t4b9\" (UID: \"fd2246d1-f0b9-4817-a935-5dbe2f9b3577\") " pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:04.325435 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.325376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728d7\" (UniqueName: \"kubernetes.io/projected/fd2246d1-f0b9-4817-a935-5dbe2f9b3577-kube-api-access-728d7\") pod \"kuadrant-operator-catalog-7t4b9\" (UID: \"fd2246d1-f0b9-4817-a935-5dbe2f9b3577\") " pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:04.332888 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.332864 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-728d7\" (UniqueName: \"kubernetes.io/projected/fd2246d1-f0b9-4817-a935-5dbe2f9b3577-kube-api-access-728d7\") pod \"kuadrant-operator-catalog-7t4b9\" (UID: \"fd2246d1-f0b9-4817-a935-5dbe2f9b3577\") " pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:04.404138 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.404114 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:04.479255 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.479221 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:04.588371 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.588262 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:04.651658 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.651621 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" event={"ID":"65c3a02e-a646-428f-b7fb-544c4860f30c","Type":"ContainerStarted","Data":"a079d71b0a3f386a979226bb0a1992ccb0a5165a994b4e3abba043621e931623"} Jul 10 17:49:04.834495 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:04.834466 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7t4b9"] Jul 10 17:49:04.870556 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:04.870452 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2246d1_f0b9_4817_a935_5dbe2f9b3577.slice/crio-01d431daf7f5247131a223d7499906763a2e8d1e5f948308efcd0b34f73c5c67 WatchSource:0}: Error finding container 01d431daf7f5247131a223d7499906763a2e8d1e5f948308efcd0b34f73c5c67: Status 404 returned error can't find the container with id 01d431daf7f5247131a223d7499906763a2e8d1e5f948308efcd0b34f73c5c67 Jul 10 17:49:05.660727 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:05.660693 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" event={"ID":"fd2246d1-f0b9-4817-a935-5dbe2f9b3577","Type":"ContainerStarted","Data":"01d431daf7f5247131a223d7499906763a2e8d1e5f948308efcd0b34f73c5c67"} Jul 10 17:49:06.644249 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.644212 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:06.665953 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.665924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" event={"ID":"65c3a02e-a646-428f-b7fb-544c4860f30c","Type":"ContainerStarted","Data":"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e"} Jul 10 17:49:06.666434 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.666034 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" podUID="65c3a02e-a646-428f-b7fb-544c4860f30c" containerName="registry-server" containerID="cri-o://0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e" gracePeriod=2 Jul 10 17:49:06.667563 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.667541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" event={"ID":"fd2246d1-f0b9-4817-a935-5dbe2f9b3577","Type":"ContainerStarted","Data":"6e82d6f525eb7ca8ee70a53a216c3ca1a5a96bd2dcc05d05f0480624383818cd"} Jul 10 17:49:06.680457 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.680420 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" podStartSLOduration=1.706172182 podStartE2EDuration="3.680409759s" totalImagesPullingTime="1.974237577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:06.679615621 +0000 UTC m=+613.740430955" watchObservedRunningTime="2026-07-10 17:49:06.680409759 +0000 UTC m=+613.741225093" Jul 10 17:49:06.693776 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.693741 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" podStartSLOduration=1.492806421 podStartE2EDuration="2.693729151s" totalImagesPullingTime="1.20092273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:06.692533487 +0000 UTC m=+613.753348823" watchObservedRunningTime="2026-07-10 17:49:06.693729151 +0000 UTC m=+613.754544485" Jul 10 17:49:06.911330 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:06.911309 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:07.052248 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.052219 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hjpd\" (UniqueName: \"kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd\") pod \"65c3a02e-a646-428f-b7fb-544c4860f30c\" (UID: \"65c3a02e-a646-428f-b7fb-544c4860f30c\") " Jul 10 17:49:07.054593 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.054565 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd" (OuterVolumeSpecName: "kube-api-access-4hjpd") pod "65c3a02e-a646-428f-b7fb-544c4860f30c" (UID: "65c3a02e-a646-428f-b7fb-544c4860f30c"). InnerVolumeSpecName "kube-api-access-4hjpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:49:07.152682 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.152661 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjpd\" (UniqueName: \"kubernetes.io/projected/65c3a02e-a646-428f-b7fb-544c4860f30c-kube-api-access-4hjpd\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:07.673187 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.673157 2582 generic.go:356] "Generic (PLEG): container finished" podID="65c3a02e-a646-428f-b7fb-544c4860f30c" containerID="0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e" exitCode=0 Jul 10 17:49:07.673569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.673199 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" Jul 10 17:49:07.673569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.673245 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" event={"ID":"65c3a02e-a646-428f-b7fb-544c4860f30c","Type":"ContainerDied","Data":"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e"} Jul 10 17:49:07.673569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.673291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2kzmr" event={"ID":"65c3a02e-a646-428f-b7fb-544c4860f30c","Type":"ContainerDied","Data":"a079d71b0a3f386a979226bb0a1992ccb0a5165a994b4e3abba043621e931623"} Jul 10 17:49:07.673569 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.673311 2582 scope.go:117] "RemoveContainer" containerID="0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e" Jul 10 17:49:07.681924 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.681904 2582 scope.go:117] "RemoveContainer" containerID="0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e" Jul 10 17:49:07.682193 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:07.682175 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e\": container with ID starting with 0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e not found: ID does not exist" containerID="0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e" Jul 10 17:49:07.682253 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.682207 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e"} err="failed to get container status \"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e\": rpc error: code = NotFound desc = could not find container \"0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e\": container with ID starting with 0a00330137c8bf8468fe2a79540ff02d1d39b4e742c2067433d4491881a03a3e not found: ID does not exist" Jul 10 17:49:07.686935 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.686915 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:07.689196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:07.689148 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2kzmr"] Jul 10 17:49:09.432188 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:09.432155 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c3a02e-a646-428f-b7fb-544c4860f30c" path="/var/lib/kubelet/pods/65c3a02e-a646-428f-b7fb-544c4860f30c/volumes" Jul 10 17:49:14.480254 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:14.480174 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:14.480254 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:14.480242 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:14.502442 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:14.502418 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:14.722864 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:14.722840 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7t4b9" Jul 10 17:49:15.775198 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:15.775167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:15.875442 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:15.875417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:22.770435 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:22.770399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:22.871257 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:22.871214 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:28.946140 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:28.946107 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:29.044949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:29.044916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:29.150607 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:29.150576 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:33.972790 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:33.972758 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:34.882302 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.882270 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp"] Jul 10 17:49:34.882591 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.882578 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c3a02e-a646-428f-b7fb-544c4860f30c" containerName="registry-server" Jul 10 17:49:34.882640 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.882592 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c3a02e-a646-428f-b7fb-544c4860f30c" containerName="registry-server" Jul 10 17:49:34.882681 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.882646 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c3a02e-a646-428f-b7fb-544c4860f30c" containerName="registry-server" Jul 10 17:49:34.885652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.885633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:34.887939 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.887916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5zmsz\"" Jul 10 17:49:34.897155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.897134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp"] Jul 10 17:49:34.974328 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.974302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:34.974676 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:34.974335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtkt\" (UniqueName: \"kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.075392 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.075356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.075504 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.075397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtkt\" (UniqueName: \"kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.075780 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.075762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.085695 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.085676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtkt\" (UniqueName: \"kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.194828 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.194799 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:35.326786 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.326708 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp"] Jul 10 17:49:35.329102 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:35.329073 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8528eb_70b2_43c2_a8dc_13af9cf570eb.slice/crio-89ce6d08cb4e2f34f7f77767d7bda30f4b7635f93a5db0f89eb11461e8c317d1 WatchSource:0}: Error finding container 89ce6d08cb4e2f34f7f77767d7bda30f4b7635f93a5db0f89eb11461e8c317d1: Status 404 returned error can't find the container with id 89ce6d08cb4e2f34f7f77767d7bda30f4b7635f93a5db0f89eb11461e8c317d1 Jul 10 17:49:35.467538 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.467451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:35.719333 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.719254 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:35.787132 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:35.787097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" event={"ID":"1f8528eb-70b2-43c2-a8dc-13af9cf570eb","Type":"ContainerStarted","Data":"89ce6d08cb4e2f34f7f77767d7bda30f4b7635f93a5db0f89eb11461e8c317d1"} Jul 10 17:49:36.082217 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:36.082144 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:38.550657 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.550626 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2696x"] Jul 10 17:49:38.554012 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.553970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:38.557078 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.557040 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zvrxs\"" Jul 10 17:49:38.564661 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.564636 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2696x"] Jul 10 17:49:38.606141 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.606112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcl77\" (UniqueName: \"kubernetes.io/projected/e1c2ba02-0516-421d-8397-c8287760f5fe-kube-api-access-hcl77\") pod \"authorino-operator-75cc8d5cb7-2696x\" (UID: \"e1c2ba02-0516-421d-8397-c8287760f5fe\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:38.707197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.707150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcl77\" (UniqueName: \"kubernetes.io/projected/e1c2ba02-0516-421d-8397-c8287760f5fe-kube-api-access-hcl77\") pod \"authorino-operator-75cc8d5cb7-2696x\" (UID: \"e1c2ba02-0516-421d-8397-c8287760f5fe\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:38.717291 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.717258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcl77\" (UniqueName: \"kubernetes.io/projected/e1c2ba02-0516-421d-8397-c8287760f5fe-kube-api-access-hcl77\") pod \"authorino-operator-75cc8d5cb7-2696x\" (UID: \"e1c2ba02-0516-421d-8397-c8287760f5fe\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:38.868364 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:38.868286 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:39.022155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:39.022130 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2696x"] Jul 10 17:49:39.024638 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:39.024608 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c2ba02_0516_421d_8397_c8287760f5fe.slice/crio-de298fe8e9973f5bb4aa29399df1cd636b1522fa73b90e0a9d664038f00c7ccd WatchSource:0}: Error finding container de298fe8e9973f5bb4aa29399df1cd636b1522fa73b90e0a9d664038f00c7ccd: Status 404 returned error can't find the container with id de298fe8e9973f5bb4aa29399df1cd636b1522fa73b90e0a9d664038f00c7ccd Jul 10 17:49:39.309250 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:39.309218 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:39.693289 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:39.693135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:39.806508 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:39.806475 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" event={"ID":"e1c2ba02-0516-421d-8397-c8287760f5fe","Type":"ContainerStarted","Data":"de298fe8e9973f5bb4aa29399df1cd636b1522fa73b90e0a9d664038f00c7ccd"} Jul 10 17:49:41.845791 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:41.845760 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg"] Jul 10 17:49:41.857215 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:41.857191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:41.859419 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:41.859397 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b5mrb\"" Jul 10 17:49:41.862142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:41.862120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg"] Jul 10 17:49:41.934232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:41.934206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrl2g\" (UniqueName: \"kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g\") pod \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" (UID: \"cbac925d-e5df-4c07-99f3-bf0f988f1609\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:42.035737 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.035701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrl2g\" (UniqueName: \"kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g\") pod \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" (UID: \"cbac925d-e5df-4c07-99f3-bf0f988f1609\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:42.045708 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.045676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrl2g\" (UniqueName: \"kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g\") pod \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" (UID: \"cbac925d-e5df-4c07-99f3-bf0f988f1609\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:42.169464 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.169441 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:42.343058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.343030 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg"] Jul 10 17:49:42.345548 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:42.345517 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbac925d_e5df_4c07_99f3_bf0f988f1609.slice/crio-bd4c4b44987b4585d98cdcf0cfa9c3ff286a96f03042ef89275222c7ccc89cbb WatchSource:0}: Error finding container bd4c4b44987b4585d98cdcf0cfa9c3ff286a96f03042ef89275222c7ccc89cbb: Status 404 returned error can't find the container with id bd4c4b44987b4585d98cdcf0cfa9c3ff286a96f03042ef89275222c7ccc89cbb Jul 10 17:49:42.500887 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.500805 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:42.710553 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.710521 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:42.822275 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.822188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" event={"ID":"1f8528eb-70b2-43c2-a8dc-13af9cf570eb","Type":"ContainerStarted","Data":"1e75faa56bc5b40027e044e08b3671ac9d3b963e86816057546c5dc8fc61f422"} Jul 10 17:49:42.822448 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.822307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:42.823868 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.823841 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" event={"ID":"e1c2ba02-0516-421d-8397-c8287760f5fe","Type":"ContainerStarted","Data":"c6012d37f5184c80311333d8559ceaca581671700a206006403744ff11146619"} Jul 10 17:49:42.823981 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.823944 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:42.824924 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.824901 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" event={"ID":"cbac925d-e5df-4c07-99f3-bf0f988f1609","Type":"ContainerStarted","Data":"bd4c4b44987b4585d98cdcf0cfa9c3ff286a96f03042ef89275222c7ccc89cbb"} Jul 10 17:49:42.843984 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.843940 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" podStartSLOduration=2.009675545 podStartE2EDuration="8.843926964s" totalImagesPullingTime="6.834251419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:42.842960957 +0000 UTC m=+649.903776289" watchObservedRunningTime="2026-07-10 17:49:42.843926964 +0000 UTC m=+649.904742364" Jul 10 17:49:42.860657 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:42.860594 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" podStartSLOduration=1.686987199 podStartE2EDuration="4.860582613s" totalImagesPullingTime="3.173595414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:42.859569052 +0000 UTC m=+649.920384402" watchObservedRunningTime="2026-07-10 17:49:42.860582613 +0000 UTC m=+649.921397947" Jul 10 17:49:44.451158 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.451122 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg"] Jul 10 17:49:44.454851 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.454832 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:44.457197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.457172 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mnckw\"" Jul 10 17:49:44.457286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.457179 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 17:49:44.465857 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.465835 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg"] Jul 10 17:49:44.555273 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.555241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9fg\" (UniqueName: \"kubernetes.io/projected/bd3c61da-cc52-4750-8304-ed54560522ca-kube-api-access-6l9fg\") pod \"dns-operator-controller-manager-864dd78c4c-cnrrg\" (UID: \"bd3c61da-cc52-4750-8304-ed54560522ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:44.656651 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.656616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9fg\" (UniqueName: \"kubernetes.io/projected/bd3c61da-cc52-4750-8304-ed54560522ca-kube-api-access-6l9fg\") pod \"dns-operator-controller-manager-864dd78c4c-cnrrg\" (UID: \"bd3c61da-cc52-4750-8304-ed54560522ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:44.675261 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.675244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9fg\" (UniqueName: \"kubernetes.io/projected/bd3c61da-cc52-4750-8304-ed54560522ca-kube-api-access-6l9fg\") pod \"dns-operator-controller-manager-864dd78c4c-cnrrg\" (UID: \"bd3c61da-cc52-4750-8304-ed54560522ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:44.767848 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.767812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:44.842067 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.842030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" event={"ID":"cbac925d-e5df-4c07-99f3-bf0f988f1609","Type":"ContainerStarted","Data":"39f4733b5c916b936d0619b1a25599187dc93bc6929a2a2441569e96668ff421"} Jul 10 17:49:44.842301 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.842286 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:44.862106 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.861692 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" podStartSLOduration=1.539093483 podStartE2EDuration="3.861670945s" totalImagesPullingTime="2.322577462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:44.861161592 +0000 UTC m=+651.921976928" watchObservedRunningTime="2026-07-10 17:49:44.861670945 +0000 UTC m=+651.922486281" Jul 10 17:49:44.904508 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:44.904483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg"] Jul 10 17:49:44.907321 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:44.907296 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3c61da_cc52_4750_8304_ed54560522ca.slice/crio-27ff02032cf5f476618ec7bf2d185d0d2b3e23b85392b211bf46055704ccb7d6 WatchSource:0}: Error finding container 27ff02032cf5f476618ec7bf2d185d0d2b3e23b85392b211bf46055704ccb7d6: Status 404 returned error can't find the container with id 27ff02032cf5f476618ec7bf2d185d0d2b3e23b85392b211bf46055704ccb7d6 Jul 10 17:49:45.350035 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:45.350009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:45.480474 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:45.480442 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:45.580491 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:45.580457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:45.712063 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:45.712025 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:45.854422 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:45.854334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" event={"ID":"bd3c61da-cc52-4750-8304-ed54560522ca","Type":"ContainerStarted","Data":"27ff02032cf5f476618ec7bf2d185d0d2b3e23b85392b211bf46055704ccb7d6"} Jul 10 17:49:46.665299 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:46.665252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:46.730880 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:46.730821 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:47.866809 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:47.866772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" event={"ID":"bd3c61da-cc52-4750-8304-ed54560522ca","Type":"ContainerStarted","Data":"e4d7dee10dd757b571e3e057545b974ac05836952aba1eb8d8e252e2cf814951"} Jul 10 17:49:47.867223 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:47.866854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:47.883229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:47.883191 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" podStartSLOduration=1.790888783 podStartE2EDuration="3.88317607s" totalImagesPullingTime="2.092287287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:47.883014011 +0000 UTC m=+654.943829339" watchObservedRunningTime="2026-07-10 17:49:47.88317607 +0000 UTC m=+654.943991406" Jul 10 17:49:53.833901 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:53.833867 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2696x" Jul 10 17:49:53.834302 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:53.833939 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.619670 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.619633 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p"] Jul 10 17:49:55.624447 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.624424 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.638069 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.638045 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p"] Jul 10 17:49:55.682458 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.682431 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp"] Jul 10 17:49:55.682602 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.682583 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" containerName="manager" containerID="cri-o://1e75faa56bc5b40027e044e08b3671ac9d3b963e86816057546c5dc8fc61f422" gracePeriod=2 Jul 10 17:49:55.694585 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.694561 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp"] Jul 10 17:49:55.702901 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.702873 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\": pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.709322 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.709300 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p"] Jul 10 17:49:55.709562 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.709545 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-xfz88], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" Jul 10 17:49:55.712267 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.712248 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:49:55.712552 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.712541 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" containerName="manager" Jul 10 17:49:55.712596 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.712554 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" containerName="manager" Jul 10 17:49:55.712629 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.712618 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" containerName="manager" Jul 10 17:49:55.715606 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.715587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.719786 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.719764 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p"] Jul 10 17:49:55.727928 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.727896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:49:55.736253 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.736232 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:49:55.739851 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.739831 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg"] Jul 10 17:49:55.739954 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.739931 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.740068 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.740048 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" containerName="manager" containerID="cri-o://39f4733b5c916b936d0619b1a25599187dc93bc6929a2a2441569e96668ff421" gracePeriod=2 Jul 10 17:49:55.742342 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.742322 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.747519 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.747500 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg"] Jul 10 17:49:55.750386 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.750204 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.750386 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.750242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfz88\" (UniqueName: \"kubernetes.io/projected/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-kube-api-access-xfz88\") pod \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.751310 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.751277 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.763362 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.756244 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:49:55.764864 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.764844 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc"] Jul 10 17:49:55.765162 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.765150 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" containerName="manager" Jul 10 17:49:55.765206 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.765164 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" containerName="manager" Jul 10 17:49:55.765259 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.765225 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" containerName="manager" Jul 10 17:49:55.768166 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.768151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:55.789321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.789296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc"] Jul 10 17:49:55.792007 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.791961 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.794364 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.794339 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.796456 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.796433 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.798642 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.798615 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.851029 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.850985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.851130 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851068 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.851130 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfz88\" (UniqueName: \"kubernetes.io/projected/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-kube-api-access-xfz88\") pod \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.851269 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851131 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwh9k\" (UniqueName: \"kubernetes.io/projected/da81e2cc-4d37-4c62-bbaf-fdc1531af550-kube-api-access-bwh9k\") pod \"limitador-operator-controller-manager-859d7b78bd-b66sc\" (UID: \"da81e2cc-4d37-4c62-bbaf-fdc1531af550\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:55.851269 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfpjk\" (UniqueName: \"kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.851269 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.851425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnflb\" (UniqueName: \"kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.851425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.851373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.857048 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.857029 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xfz88 for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-s9v8p" not found Jul 10 17:49:55.857172 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.857095 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-kube-api-access-xfz88 podName:b827aebb-7fe9-4401-9ed1-f9cbffa1247f nodeName:}" failed. No retries permitted until 2026-07-10 17:49:56.357074477 +0000 UTC m=+663.417889790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xfz88" (UniqueName: "kubernetes.io/projected/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-kube-api-access-xfz88") pod "kuadrant-operator-controller-manager-86878c769f-s9v8p" (UID: "b827aebb-7fe9-4401-9ed1-f9cbffa1247f") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-s9v8p" not found Jul 10 17:49:55.904229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.904188 2582 generic.go:356] "Generic (PLEG): container finished" podID="cbac925d-e5df-4c07-99f3-bf0f988f1609" containerID="39f4733b5c916b936d0619b1a25599187dc93bc6929a2a2441569e96668ff421" exitCode=0 Jul 10 17:49:55.906272 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.906247 2582 generic.go:356] "Generic (PLEG): container finished" podID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" containerID="1e75faa56bc5b40027e044e08b3671ac9d3b963e86816057546c5dc8fc61f422" exitCode=0 Jul 10 17:49:55.906414 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.906401 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.909254 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.909221 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.913340 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.913316 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.914379 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.914356 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.919124 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.919094 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.921714 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.921677 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.923707 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.923679 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.925674 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.925649 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.952197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952174 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfpjk\" (UniqueName: \"kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.952306 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.952306 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnflb\" (UniqueName: \"kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.952426 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.952426 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwh9k\" (UniqueName: \"kubernetes.io/projected/da81e2cc-4d37-4c62-bbaf-fdc1531af550-kube-api-access-bwh9k\") pod \"limitador-operator-controller-manager-859d7b78bd-b66sc\" (UID: \"da81e2cc-4d37-4c62-bbaf-fdc1531af550\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:55.952645 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952623 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.952815 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.952793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.957239 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.957207 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:55.963165 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.963145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnflb\" (UniqueName: \"kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb\") pod \"kuadrant-operator-controller-manager-86878c769f-zbk7v\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:55.963263 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.963198 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwh9k\" (UniqueName: \"kubernetes.io/projected/da81e2cc-4d37-4c62-bbaf-fdc1531af550-kube-api-access-bwh9k\") pod \"limitador-operator-controller-manager-859d7b78bd-b66sc\" (UID: \"da81e2cc-4d37-4c62-bbaf-fdc1531af550\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:55.963383 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.963360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfpjk\" (UniqueName: \"kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6jwjk\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:55.991389 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.991366 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.993345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:55.993329 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.993629 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.993610 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:55.995496 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.995475 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:55.997304 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.997286 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:55.999144 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:55.999124 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:56.000963 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.000945 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:56.002789 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.002767 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:56.053145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053110 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrl2g\" (UniqueName: \"kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g\") pod \"cbac925d-e5df-4c07-99f3-bf0f988f1609\" (UID: \"cbac925d-e5df-4c07-99f3-bf0f988f1609\") " Jul 10 17:49:56.053145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053153 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume\") pod \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " Jul 10 17:49:56.053313 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume\") pod \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\" (UID: \"b827aebb-7fe9-4401-9ed1-f9cbffa1247f\") " Jul 10 17:49:56.053313 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053301 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dtkt\" (UniqueName: \"kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt\") pod \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\" (UID: \"1f8528eb-70b2-43c2-a8dc-13af9cf570eb\") " Jul 10 17:49:56.053411 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053395 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b827aebb-7fe9-4401-9ed1-f9cbffa1247f" (UID: "b827aebb-7fe9-4401-9ed1-f9cbffa1247f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:49:56.053529 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053505 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1f8528eb-70b2-43c2-a8dc-13af9cf570eb" (UID: "1f8528eb-70b2-43c2-a8dc-13af9cf570eb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:49:56.053637 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053572 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-extensions-socket-volume\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:56.053637 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053587 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfz88\" (UniqueName: \"kubernetes.io/projected/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-kube-api-access-xfz88\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:56.053637 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.053596 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b827aebb-7fe9-4401-9ed1-f9cbffa1247f-extensions-socket-volume\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:56.055298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.055278 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g" (OuterVolumeSpecName: "kube-api-access-jrl2g") pod "cbac925d-e5df-4c07-99f3-bf0f988f1609" (UID: "cbac925d-e5df-4c07-99f3-bf0f988f1609"). InnerVolumeSpecName "kube-api-access-jrl2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:49:56.055363 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.055323 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt" (OuterVolumeSpecName: "kube-api-access-2dtkt") pod "1f8528eb-70b2-43c2-a8dc-13af9cf570eb" (UID: "1f8528eb-70b2-43c2-a8dc-13af9cf570eb"). InnerVolumeSpecName "kube-api-access-2dtkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:49:56.056880 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.056859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:56.109006 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.108969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:56.116934 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.116917 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:56.121918 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.121900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:56.154171 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.154134 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrl2g\" (UniqueName: \"kubernetes.io/projected/cbac925d-e5df-4c07-99f3-bf0f988f1609-kube-api-access-jrl2g\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:56.154171 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.154172 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtkt\" (UniqueName: \"kubernetes.io/projected/1f8528eb-70b2-43c2-a8dc-13af9cf570eb-kube-api-access-2dtkt\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:49:56.260534 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.260504 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:49:56.264001 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:56.263963 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b2bf8e_34b9_4b87_a58c_53140d1fdb14.slice/crio-58894e61139b7d16e34d567a6627881e08131fc4112b4623b8a298b949d072db WatchSource:0}: Error finding container 58894e61139b7d16e34d567a6627881e08131fc4112b4623b8a298b949d072db: Status 404 returned error can't find the container with id 58894e61139b7d16e34d567a6627881e08131fc4112b4623b8a298b949d072db Jul 10 17:49:56.491931 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.491873 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc"] Jul 10 17:49:56.494623 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.494598 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:49:56.494847 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:56.494823 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda81e2cc_4d37_4c62_bbaf_fdc1531af550.slice/crio-a8a7c0eb570c9170616c22623f3b4d9a86df2f85fc0cf823112deb109df88e3f WatchSource:0}: Error finding container a8a7c0eb570c9170616c22623f3b4d9a86df2f85fc0cf823112deb109df88e3f: Status 404 returned error can't find the container with id a8a7c0eb570c9170616c22623f3b4d9a86df2f85fc0cf823112deb109df88e3f Jul 10 17:49:56.497358 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:49:56.497320 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7449cdf1_b8e1_4539_82b7_0b2d256c383d.slice/crio-db122e6577e00e5d1bb0d138a158b011b9a392f31066b0dd45387e75b43caa22 WatchSource:0}: Error finding container db122e6577e00e5d1bb0d138a158b011b9a392f31066b0dd45387e75b43caa22: Status 404 returned error can't find the container with id db122e6577e00e5d1bb0d138a158b011b9a392f31066b0dd45387e75b43caa22 Jul 10 17:49:56.912725 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.912697 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" event={"ID":"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14","Type":"ContainerStarted","Data":"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4"} Jul 10 17:49:56.913200 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.912745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" event={"ID":"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14","Type":"ContainerStarted","Data":"58894e61139b7d16e34d567a6627881e08131fc4112b4623b8a298b949d072db"} Jul 10 17:49:56.913200 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.912788 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:49:56.914291 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.914275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:56.914291 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.914285 2582 scope.go:117] "RemoveContainer" containerID="39f4733b5c916b936d0619b1a25599187dc93bc6929a2a2441569e96668ff421" Jul 10 17:49:56.916412 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.916388 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" event={"ID":"da81e2cc-4d37-4c62-bbaf-fdc1531af550","Type":"ContainerStarted","Data":"181b0e451fb89460c15cd279537cc1151d56bc189415ebd4139f50a3694701cc"} Jul 10 17:49:56.916484 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.916439 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:49:56.916532 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.916503 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" event={"ID":"da81e2cc-4d37-4c62-bbaf-fdc1531af550","Type":"ContainerStarted","Data":"a8a7c0eb570c9170616c22623f3b4d9a86df2f85fc0cf823112deb109df88e3f"} Jul 10 17:49:56.918294 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.918274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" event={"ID":"7449cdf1-b8e1-4539-82b7-0b2d256c383d","Type":"ContainerStarted","Data":"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af"} Jul 10 17:49:56.918407 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.918315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:49:56.918407 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.918338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" event={"ID":"7449cdf1-b8e1-4539-82b7-0b2d256c383d","Type":"ContainerStarted","Data":"db122e6577e00e5d1bb0d138a158b011b9a392f31066b0dd45387e75b43caa22"} Jul 10 17:49:56.920342 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.919795 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:56.920342 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.919804 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:56.923696 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.923591 2582 scope.go:117] "RemoveContainer" containerID="1e75faa56bc5b40027e044e08b3671ac9d3b963e86816057546c5dc8fc61f422" Jul 10 17:49:56.946958 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.946915 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" podStartSLOduration=1.946902418 podStartE2EDuration="1.946902418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:56.945049398 +0000 UTC m=+664.005864732" watchObservedRunningTime="2026-07-10 17:49:56.946902418 +0000 UTC m=+664.007717820" Jul 10 17:49:56.947444 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.947421 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:56.949762 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.949731 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:56.951925 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.951891 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:56.987908 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:56.987870 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" podStartSLOduration=1.987858456 podStartE2EDuration="1.987858456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:56.986650073 +0000 UTC m=+664.047465407" watchObservedRunningTime="2026-07-10 17:49:56.987858456 +0000 UTC m=+664.048673824" Jul 10 17:49:56.989431 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:56.989406 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-s9v8p\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s9v8p" Jul 10 17:49:57.019196 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:57.019150 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" podStartSLOduration=2.019115396 podStartE2EDuration="2.019115396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:49:57.01733345 +0000 UTC m=+664.078148808" watchObservedRunningTime="2026-07-10 17:49:57.019115396 +0000 UTC m=+664.079930730" Jul 10 17:49:57.019507 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:57.019483 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c6qvp\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c6qvp" Jul 10 17:49:57.021454 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:49:57.021430 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cxcfg\" is forbidden: User \"system:node:ip-10-0-133-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-236.ec2.internal' and this object" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cxcfg" Jul 10 17:49:57.433221 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:57.433189 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8528eb-70b2-43c2-a8dc-13af9cf570eb" path="/var/lib/kubelet/pods/1f8528eb-70b2-43c2-a8dc-13af9cf570eb/volumes" Jul 10 17:49:57.433554 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:57.433539 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b827aebb-7fe9-4401-9ed1-f9cbffa1247f" path="/var/lib/kubelet/pods/b827aebb-7fe9-4401-9ed1-f9cbffa1247f/volumes" Jul 10 17:49:57.433758 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:57.433746 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbac925d-e5df-4c07-99f3-bf0f988f1609" path="/var/lib/kubelet/pods/cbac925d-e5df-4c07-99f3-bf0f988f1609/volumes" Jul 10 17:49:58.873013 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:58.872965 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cnrrg" Jul 10 17:49:58.978685 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:58.978650 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:49:59.076155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:49:59.076123 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:01.339648 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:01.339613 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:01.440948 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:01.440917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:04.008422 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:04.008380 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:04.064303 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:04.064253 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:06.878001 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:06.877959 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:06.932670 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:06.932625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:07.928592 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:07.928565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-b66sc" Jul 10 17:50:07.929058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:07.928741 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:50:07.929058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:07.928795 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:50:08.016479 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.016450 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:50:08.016700 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.016628 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" podUID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" containerName="manager" containerID="cri-o://0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4" gracePeriod=10 Jul 10 17:50:08.269165 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.269141 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:50:08.349103 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.349074 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfpjk\" (UniqueName: \"kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk\") pod \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " Jul 10 17:50:08.349258 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.349155 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume\") pod \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\" (UID: \"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14\") " Jul 10 17:50:08.349553 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.349528 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" (UID: "d6b2bf8e-34b9-4b87-a58c-53140d1fdb14"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:50:08.351195 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.351176 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk" (OuterVolumeSpecName: "kube-api-access-rfpjk") pod "d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" (UID: "d6b2bf8e-34b9-4b87-a58c-53140d1fdb14"). InnerVolumeSpecName "kube-api-access-rfpjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:50:08.450060 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.449967 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-extensions-socket-volume\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:50:08.450060 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.450008 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfpjk\" (UniqueName: \"kubernetes.io/projected/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14-kube-api-access-rfpjk\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:50:08.977197 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.977162 2582 generic.go:356] "Generic (PLEG): container finished" podID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" containerID="0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4" exitCode=0 Jul 10 17:50:08.977652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.977209 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" Jul 10 17:50:08.977652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.977246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" event={"ID":"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14","Type":"ContainerDied","Data":"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4"} Jul 10 17:50:08.977652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.977298 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk" event={"ID":"d6b2bf8e-34b9-4b87-a58c-53140d1fdb14","Type":"ContainerDied","Data":"58894e61139b7d16e34d567a6627881e08131fc4112b4623b8a298b949d072db"} Jul 10 17:50:08.977652 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.977320 2582 scope.go:117] "RemoveContainer" containerID="0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4" Jul 10 17:50:08.986029 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.986014 2582 scope.go:117] "RemoveContainer" containerID="0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4" Jul 10 17:50:08.986330 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:50:08.986313 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4\": container with ID starting with 0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4 not found: ID does not exist" containerID="0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4" Jul 10 17:50:08.986396 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.986338 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4"} err="failed to get container status \"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4\": rpc error: code = NotFound desc = could not find container \"0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4\": container with ID starting with 0128dc459b78c4705f1ec2c1a5367689ce4fa74d79f284fe850471573003e0c4 not found: ID does not exist" Jul 10 17:50:08.999007 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:08.998969 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:50:09.012538 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:09.012512 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6jwjk"] Jul 10 17:50:09.432303 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:09.432273 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" path="/var/lib/kubelet/pods/d6b2bf8e-34b9-4b87-a58c-53140d1fdb14/volumes" Jul 10 17:50:15.607958 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:15.607917 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:15.703446 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:15.703406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:18.609986 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:18.609947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:18.710634 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:18.710600 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:21.038138 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:21.038103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:21.090045 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:21.090023 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:22.021093 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:22.021063 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:22.122903 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:22.122874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:24.368831 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.368800 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj"] Jul 10 17:50:24.369225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.369212 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" containerName="manager" Jul 10 17:50:24.369225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.369225 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" containerName="manager" Jul 10 17:50:24.369303 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.369280 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b2bf8e-34b9-4b87-a58c-53140d1fdb14" containerName="manager" Jul 10 17:50:24.375243 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.375224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.377488 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.377462 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vrcxr\"" Jul 10 17:50:24.384369 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.384347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj"] Jul 10 17:50:24.391919 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.391896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:24.480144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480250 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480286 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480571 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f28aaa-bf3a-4b18-bc39-42f789367991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480571 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khnds\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-kube-api-access-khnds\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.480571 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.480429 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.491824 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.491795 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:24.581548 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581706 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581706 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581815 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581815 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581815 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581790 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581815 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f28aaa-bf3a-4b18-bc39-42f789367991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khnds\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-kube-api-access-khnds\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.581966 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.581922 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.582252 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.582077 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.582252 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.582179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.582369 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.582246 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.582543 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.582526 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f28aaa-bf3a-4b18-bc39-42f789367991-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.583973 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.583948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.584182 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.584166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.589205 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.589182 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.589475 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.589456 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khnds\" (UniqueName: \"kubernetes.io/projected/c3f28aaa-bf3a-4b18-bc39-42f789367991-kube-api-access-khnds\") pod \"maas-default-gateway-openshift-default-5896d4dd49-jhpkj\" (UID: \"c3f28aaa-bf3a-4b18-bc39-42f789367991\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.686317 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.686284 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:24.824160 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.824134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj"] Jul 10 17:50:24.826215 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:50:24.826190 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f28aaa_bf3a_4b18_bc39_42f789367991.slice/crio-225dfa58995b6738fc752890e70bd88adc240de4433901b3da983e9192686f84 WatchSource:0}: Error finding container 225dfa58995b6738fc752890e70bd88adc240de4433901b3da983e9192686f84: Status 404 returned error can't find the container with id 225dfa58995b6738fc752890e70bd88adc240de4433901b3da983e9192686f84 Jul 10 17:50:24.828453 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.828419 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:50:24.828551 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.828499 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:50:24.828551 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:24.828544 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:50:25.050616 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:25.050511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" event={"ID":"c3f28aaa-bf3a-4b18-bc39-42f789367991","Type":"ContainerStarted","Data":"77b33f38fd30d3fc01bd89a2fbaad958b933204f9edde18cdb4c1ca7f6dfe6bb"} Jul 10 17:50:25.050616 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:25.050571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" event={"ID":"c3f28aaa-bf3a-4b18-bc39-42f789367991","Type":"ContainerStarted","Data":"225dfa58995b6738fc752890e70bd88adc240de4433901b3da983e9192686f84"} Jul 10 17:50:25.070589 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:25.070537 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" podStartSLOduration=1.07052289 podStartE2EDuration="1.07052289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:50:25.069659248 +0000 UTC m=+692.130474583" watchObservedRunningTime="2026-07-10 17:50:25.07052289 +0000 UTC m=+692.131338225" Jul 10 17:50:25.686731 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:25.686698 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:25.691461 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:25.691436 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:26.051668 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.051597 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:26.055592 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.055570 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:26.056553 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.056539 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:50:26.173734 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.173704 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:26.274123 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.274033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:26.858732 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.858701 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:26.862341 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.862322 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:26.864847 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.864825 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xs9m\"" Jul 10 17:50:26.873034 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.873013 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:26.899058 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.899034 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:26.956110 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:26.956080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:27.002527 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.002506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zpcp\" (UniqueName: \"kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp\") pod \"authorino-58cd955675-9tr6h\" (UID: \"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964\") " pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:27.036011 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.035969 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:27.099707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.099679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:27.104038 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.103986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zpcp\" (UniqueName: \"kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp\") pod \"authorino-58cd955675-9tr6h\" (UID: \"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964\") " pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:27.112164 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.112109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zpcp\" (UniqueName: \"kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp\") pod \"authorino-58cd955675-9tr6h\" (UID: \"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964\") " pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:27.171918 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.171900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:27.302773 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.302751 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:27.304641 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:50:27.304616 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5be7be3_cc6d_404d_aff8_ba5a5c0fa964.slice/crio-b1c9f6463c18169fcf75598814e95be266bbcbaac5a34aeb953ca5c75d3063d2 WatchSource:0}: Error finding container b1c9f6463c18169fcf75598814e95be266bbcbaac5a34aeb953ca5c75d3063d2: Status 404 returned error can't find the container with id b1c9f6463c18169fcf75598814e95be266bbcbaac5a34aeb953ca5c75d3063d2 Jul 10 17:50:27.306132 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:27.306116 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:50:28.069920 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:28.069883 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9tr6h" event={"ID":"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964","Type":"ContainerStarted","Data":"b1c9f6463c18169fcf75598814e95be266bbcbaac5a34aeb953ca5c75d3063d2"} Jul 10 17:50:31.088442 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:31.088406 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9tr6h" event={"ID":"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964","Type":"ContainerStarted","Data":"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3"} Jul 10 17:50:31.105954 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:31.105907 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9tr6h" podStartSLOduration=2.051247487 podStartE2EDuration="5.105892251s" totalImagesPullingTime="3.054644764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:50:31.104526494 +0000 UTC m=+698.165341827" watchObservedRunningTime="2026-07-10 17:50:31.105892251 +0000 UTC m=+698.166707586" Jul 10 17:50:32.183849 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:32.183814 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:33.097026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:33.096970 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9tr6h" podUID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" containerName="authorino" containerID="cri-o://fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3" gracePeriod=30 Jul 10 17:50:33.340422 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:33.340395 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:33.357833 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:33.357771 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zpcp\" (UniqueName: \"kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp\") pod \"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964\" (UID: \"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964\") " Jul 10 17:50:33.360136 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:33.360112 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp" (OuterVolumeSpecName: "kube-api-access-2zpcp") pod "e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" (UID: "e5be7be3-cc6d-404d-aff8-ba5a5c0fa964"). InnerVolumeSpecName "kube-api-access-2zpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:50:33.459160 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:33.459133 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zpcp\" (UniqueName: \"kubernetes.io/projected/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964-kube-api-access-2zpcp\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:50:34.102911 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.102870 2582 generic.go:356] "Generic (PLEG): container finished" podID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" containerID="fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3" exitCode=0 Jul 10 17:50:34.102911 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.102910 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9tr6h" Jul 10 17:50:34.103155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.102955 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9tr6h" event={"ID":"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964","Type":"ContainerDied","Data":"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3"} Jul 10 17:50:34.103155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.103013 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9tr6h" event={"ID":"e5be7be3-cc6d-404d-aff8-ba5a5c0fa964","Type":"ContainerDied","Data":"b1c9f6463c18169fcf75598814e95be266bbcbaac5a34aeb953ca5c75d3063d2"} Jul 10 17:50:34.103155 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.103030 2582 scope.go:117] "RemoveContainer" containerID="fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3" Jul 10 17:50:34.111444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.111427 2582 scope.go:117] "RemoveContainer" containerID="fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3" Jul 10 17:50:34.111678 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:50:34.111659 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3\": container with ID starting with fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3 not found: ID does not exist" containerID="fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3" Jul 10 17:50:34.111747 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.111689 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3"} err="failed to get container status \"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3\": rpc error: code = NotFound desc = could not find container \"fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3\": container with ID starting with fcd103b86fc8f317646673a6cab2d35317ef74403bb401aa36e6dc8bbd9727b3 not found: ID does not exist" Jul 10 17:50:34.118173 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.118149 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:34.121555 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:34.121532 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9tr6h"] Jul 10 17:50:35.432082 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:35.432047 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" path="/var/lib/kubelet/pods/e5be7be3-cc6d-404d-aff8-ba5a5c0fa964/volumes" Jul 10 17:50:35.816765 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:35.816681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:35.913642 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:35.913615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:38.822345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:38.822307 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:38.924240 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:38.924203 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:41.267144 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:41.267112 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:41.327563 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:41.327531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:47.318140 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:47.318054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:47.424137 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:47.424103 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:50:47.791050 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:50:47.791017 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:00.043625 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.043584 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:00.272119 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.272090 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:00.272443 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.272431 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" containerName="authorino" Jul 10 17:51:00.272490 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.272445 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" containerName="authorino" Jul 10 17:51:00.272524 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.272509 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5be7be3-cc6d-404d-aff8-ba5a5c0fa964" containerName="authorino" Jul 10 17:51:00.294611 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.294537 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:00.294814 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.294624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:00.297156 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.297070 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xs9m\"" Jul 10 17:51:00.377229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.377205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8vv\" (UniqueName: \"kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv\") pod \"authorino-5d4b67c446-p6rwh\" (UID: \"de2351f1-5738-46ae-ba3e-7733c0c9bbc6\") " pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:00.379473 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.379449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:00.478104 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.478072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8vv\" (UniqueName: \"kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv\") pod \"authorino-5d4b67c446-p6rwh\" (UID: \"de2351f1-5738-46ae-ba3e-7733c0c9bbc6\") " pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:00.486154 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.486133 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8vv\" (UniqueName: \"kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv\") pod \"authorino-5d4b67c446-p6rwh\" (UID: \"de2351f1-5738-46ae-ba3e-7733c0c9bbc6\") " pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:00.509411 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.509391 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:00.509624 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.509613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:00.531132 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.531110 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fd75b984f-w4s7d"] Jul 10 17:51:00.542830 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.542804 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd75b984f-w4s7d"] Jul 10 17:51:00.542949 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.542934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:00.591488 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.591461 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd75b984f-w4s7d"] Jul 10 17:51:00.591829 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:51:00.591801 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lb4vc], unattached volumes=[], failed to process volumes=[kube-api-access-lb4vc]: context canceled" pod="kuadrant-system/authorino-5fd75b984f-w4s7d" podUID="b4a73b36-29f6-4690-a4ee-2adb25c5e273" Jul 10 17:51:00.617810 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.617724 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:51:00.648711 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.648684 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:51:00.648882 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.648790 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.651190 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.651157 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 17:51:00.667191 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.667169 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:00.669761 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:51:00.669700 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde2351f1_5738_46ae_ba3e_7733c0c9bbc6.slice/crio-2756819e9ec15e416eddfd3e4500c47147f73526f223d11546899a801c9ec000 WatchSource:0}: Error finding container 2756819e9ec15e416eddfd3e4500c47147f73526f223d11546899a801c9ec000: Status 404 returned error can't find the container with id 2756819e9ec15e416eddfd3e4500c47147f73526f223d11546899a801c9ec000 Jul 10 17:51:00.680207 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.680183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4vc\" (UniqueName: \"kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc\") pod \"authorino-5fd75b984f-w4s7d\" (UID: \"b4a73b36-29f6-4690-a4ee-2adb25c5e273\") " pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:00.781480 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.781450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djphw\" (UniqueName: \"kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.781605 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.781491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4vc\" (UniqueName: \"kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc\") pod \"authorino-5fd75b984f-w4s7d\" (UID: \"b4a73b36-29f6-4690-a4ee-2adb25c5e273\") " pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:00.781605 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.781516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.789351 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.789331 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4vc\" (UniqueName: \"kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc\") pod \"authorino-5fd75b984f-w4s7d\" (UID: \"b4a73b36-29f6-4690-a4ee-2adb25c5e273\") " pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:00.882314 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.882228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.882482 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.882351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djphw\" (UniqueName: \"kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.884772 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.884752 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.889783 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.889766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djphw\" (UniqueName: \"kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw\") pod \"authorino-7f76fc5985-mpzk9\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:00.958812 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:00.958785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:51:01.118616 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.118594 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:51:01.120792 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:51:01.120765 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfff7908_36e4_4a1d_bacd_b5303e4c4154.slice/crio-6f8105a164ebf05095088a8bbeb525a3f4addfcedf018c76a64c89f76755c6a2 WatchSource:0}: Error finding container 6f8105a164ebf05095088a8bbeb525a3f4addfcedf018c76a64c89f76755c6a2: Status 404 returned error can't find the container with id 6f8105a164ebf05095088a8bbeb525a3f4addfcedf018c76a64c89f76755c6a2 Jul 10 17:51:01.225190 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.225148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" event={"ID":"de2351f1-5738-46ae-ba3e-7733c0c9bbc6","Type":"ContainerStarted","Data":"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3"} Jul 10 17:51:01.225386 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.225195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" event={"ID":"de2351f1-5738-46ae-ba3e-7733c0c9bbc6","Type":"ContainerStarted","Data":"2756819e9ec15e416eddfd3e4500c47147f73526f223d11546899a801c9ec000"} Jul 10 17:51:01.225386 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.225249 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" podUID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" containerName="authorino" containerID="cri-o://50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3" gracePeriod=30 Jul 10 17:51:01.226560 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.226542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:01.226560 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.226541 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" event={"ID":"cfff7908-36e4-4a1d-bacd-b5303e4c4154","Type":"ContainerStarted","Data":"6f8105a164ebf05095088a8bbeb525a3f4addfcedf018c76a64c89f76755c6a2"} Jul 10 17:51:01.233157 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.233139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:01.242678 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.242639 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" podStartSLOduration=0.83447933 podStartE2EDuration="1.242625514s" totalImagesPullingTime="408.146184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:51:01.241647114 +0000 UTC m=+728.302462450" watchObservedRunningTime="2026-07-10 17:51:01.242625514 +0000 UTC m=+728.303440848" Jul 10 17:51:01.285444 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.285416 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb4vc\" (UniqueName: \"kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc\") pod \"b4a73b36-29f6-4690-a4ee-2adb25c5e273\" (UID: \"b4a73b36-29f6-4690-a4ee-2adb25c5e273\") " Jul 10 17:51:01.287460 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.287439 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc" (OuterVolumeSpecName: "kube-api-access-lb4vc") pod "b4a73b36-29f6-4690-a4ee-2adb25c5e273" (UID: "b4a73b36-29f6-4690-a4ee-2adb25c5e273"). InnerVolumeSpecName "kube-api-access-lb4vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:51:01.386753 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.386673 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb4vc\" (UniqueName: \"kubernetes.io/projected/b4a73b36-29f6-4690-a4ee-2adb25c5e273-kube-api-access-lb4vc\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:51:01.470298 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.470278 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:01.588768 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.588737 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w8vv\" (UniqueName: \"kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv\") pod \"de2351f1-5738-46ae-ba3e-7733c0c9bbc6\" (UID: \"de2351f1-5738-46ae-ba3e-7733c0c9bbc6\") " Jul 10 17:51:01.591019 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.590967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv" (OuterVolumeSpecName: "kube-api-access-9w8vv") pod "de2351f1-5738-46ae-ba3e-7733c0c9bbc6" (UID: "de2351f1-5738-46ae-ba3e-7733c0c9bbc6"). InnerVolumeSpecName "kube-api-access-9w8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:51:01.689951 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:01.689926 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w8vv\" (UniqueName: \"kubernetes.io/projected/de2351f1-5738-46ae-ba3e-7733c0c9bbc6-kube-api-access-9w8vv\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:51:02.232531 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.232496 2582 generic.go:356] "Generic (PLEG): container finished" podID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" containerID="50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3" exitCode=0 Jul 10 17:51:02.233026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.232538 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" Jul 10 17:51:02.233026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.232584 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" event={"ID":"de2351f1-5738-46ae-ba3e-7733c0c9bbc6","Type":"ContainerDied","Data":"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3"} Jul 10 17:51:02.233026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.232632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-p6rwh" event={"ID":"de2351f1-5738-46ae-ba3e-7733c0c9bbc6","Type":"ContainerDied","Data":"2756819e9ec15e416eddfd3e4500c47147f73526f223d11546899a801c9ec000"} Jul 10 17:51:02.233026 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.232656 2582 scope.go:117] "RemoveContainer" containerID="50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3" Jul 10 17:51:02.234460 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.234429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" event={"ID":"cfff7908-36e4-4a1d-bacd-b5303e4c4154","Type":"ContainerStarted","Data":"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd"} Jul 10 17:51:02.234460 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.234458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd75b984f-w4s7d" Jul 10 17:51:02.241392 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.241378 2582 scope.go:117] "RemoveContainer" containerID="50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3" Jul 10 17:51:02.241671 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:51:02.241647 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3\": container with ID starting with 50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3 not found: ID does not exist" containerID="50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3" Jul 10 17:51:02.241762 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.241681 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3"} err="failed to get container status \"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3\": rpc error: code = NotFound desc = could not find container \"50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3\": container with ID starting with 50b2c7071eacb8655c71639f8d201954e5a329dacc711a3f3a90b7c6490f9da3 not found: ID does not exist" Jul 10 17:51:02.253425 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.253386 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" podStartSLOduration=1.876624931 podStartE2EDuration="2.253375055s" totalImagesPullingTime="376.750124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:51:02.25162913 +0000 UTC m=+729.312444466" watchObservedRunningTime="2026-07-10 17:51:02.253375055 +0000 UTC m=+729.314190431" Jul 10 17:51:02.279017 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.278970 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd75b984f-w4s7d"] Jul 10 17:51:02.283038 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.283012 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fd75b984f-w4s7d"] Jul 10 17:51:02.297474 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.297449 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:02.303142 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:02.303122 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-p6rwh"] Jul 10 17:51:03.432376 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:03.432344 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a73b36-29f6-4690-a4ee-2adb25c5e273" path="/var/lib/kubelet/pods/b4a73b36-29f6-4690-a4ee-2adb25c5e273/volumes" Jul 10 17:51:03.432744 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:03.432568 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" path="/var/lib/kubelet/pods/de2351f1-5738-46ae-ba3e-7733c0c9bbc6/volumes" Jul 10 17:51:06.587371 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:06.587339 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:06.683065 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:06.683030 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:06.737476 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:06.737437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:07.063812 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:07.063785 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:10.021583 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:10.021555 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:10.120574 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:10.120549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:21.609305 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:21.609276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:42.786270 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:42.786233 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:43.206696 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.206664 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:43.661408 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.661379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:43.764577 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.764539 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:43.768702 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.768674 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9p28h"] Jul 10 17:51:43.769048 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.769035 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" containerName="authorino" Jul 10 17:51:43.769107 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.769050 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" containerName="authorino" Jul 10 17:51:43.769149 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.769114 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2351f1-5738-46ae-ba3e-7733c0c9bbc6" containerName="authorino" Jul 10 17:51:43.772234 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.772217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.774498 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.774479 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 17:51:43.774596 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.774516 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-js2vj\"" Jul 10 17:51:43.781466 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.781441 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9p28h"] Jul 10 17:51:43.860968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.860938 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.861385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.861043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvjh\" (UniqueName: \"kubernetes.io/projected/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-kube-api-access-trvjh\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.961025 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.960927 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:43.962133 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.962112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.962188 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.962166 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvjh\" (UniqueName: \"kubernetes.io/projected/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-kube-api-access-trvjh\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.962832 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.962815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-plugins-config-volume\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:43.969770 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:43.969744 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvjh\" (UniqueName: \"kubernetes.io/projected/0dec4d25-f353-4ec3-94c6-dbf3d606d30d-kube-api-access-trvjh\") pod \"payload-pre-processing-7964457696-9p28h\" (UID: \"0dec4d25-f353-4ec3-94c6-dbf3d606d30d\") " pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:44.083280 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:44.083248 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:44.214125 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:44.214101 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-9p28h"] Jul 10 17:51:44.216526 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:51:44.216498 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dec4d25_f353_4ec3_94c6_dbf3d606d30d.slice/crio-683ef4a3a88e61612dd9318514f879f5f9e88f2e74b4e1d1359f5a36e01d4927 WatchSource:0}: Error finding container 683ef4a3a88e61612dd9318514f879f5f9e88f2e74b4e1d1359f5a36e01d4927: Status 404 returned error can't find the container with id 683ef4a3a88e61612dd9318514f879f5f9e88f2e74b4e1d1359f5a36e01d4927 Jul 10 17:51:44.428265 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:44.428175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" event={"ID":"0dec4d25-f353-4ec3-94c6-dbf3d606d30d","Type":"ContainerStarted","Data":"683ef4a3a88e61612dd9318514f879f5f9e88f2e74b4e1d1359f5a36e01d4927"} Jul 10 17:51:46.439885 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:46.439846 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" event={"ID":"0dec4d25-f353-4ec3-94c6-dbf3d606d30d","Type":"ContainerStarted","Data":"c5991f24da82be9b53759748f2dd50967447b9eaca467d6961c626a4adbb8e52"} Jul 10 17:51:46.440321 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:46.439933 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:51:46.455130 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:46.455074 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" podStartSLOduration=1.945981791 podStartE2EDuration="3.455056864s" totalImagesPullingTime="1.509075073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:51:46.454809001 +0000 UTC m=+773.515624336" watchObservedRunningTime="2026-07-10 17:51:46.455056864 +0000 UTC m=+773.515872200" Jul 10 17:51:51.463930 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.463896 2582 generic.go:356] "Generic (PLEG): container finished" podID="c3f28aaa-bf3a-4b18-bc39-42f789367991" containerID="77b33f38fd30d3fc01bd89a2fbaad958b933204f9edde18cdb4c1ca7f6dfe6bb" exitCode=0 Jul 10 17:51:51.464446 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.463977 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" event={"ID":"c3f28aaa-bf3a-4b18-bc39-42f789367991","Type":"ContainerDied","Data":"77b33f38fd30d3fc01bd89a2fbaad958b933204f9edde18cdb4c1ca7f6dfe6bb"} Jul 10 17:51:51.464512 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.464490 2582 scope.go:117] "RemoveContainer" containerID="77b33f38fd30d3fc01bd89a2fbaad958b933204f9edde18cdb4c1ca7f6dfe6bb" Jul 10 17:51:51.465478 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.465450 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:51:51.465557 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.465542 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:51:51.465594 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:51.465581 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 17:51:52.470969 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:52.470934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" event={"ID":"c3f28aaa-bf3a-4b18-bc39-42f789367991","Type":"ContainerStarted","Data":"0c0ff5c74f496dec02cd0a24a27e389bbbcc86774d874ff1c69eda5fe1e20fa3"} Jul 10 17:51:52.686738 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:52.686704 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:51:52.687487 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:52.687465 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jhpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 17:51:52.687586 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:52.687512 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" podUID="c3f28aaa-bf3a-4b18-bc39-42f789367991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 17:51:53.686900 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:53.686865 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jhpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 17:51:53.687258 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:53.686944 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" podUID="c3f28aaa-bf3a-4b18-bc39-42f789367991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 17:51:53.990211 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:53.990127 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:54.094525 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:54.094457 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:51:54.686653 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:54.686613 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jhpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 17:51:54.686831 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:54.686675 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" podUID="c3f28aaa-bf3a-4b18-bc39-42f789367991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 17:51:55.687664 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:55.687628 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-jhpkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 10 17:51:55.688106 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:55.687692 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" podUID="c3f28aaa-bf3a-4b18-bc39-42f789367991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 10 17:51:56.056388 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:56.056309 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:51:56.690823 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:56.690792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:51:56.691693 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:56.691673 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-jhpkj" Jul 10 17:51:57.446684 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:51:57.446653 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-9p28h" Jul 10 17:52:00.270177 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:00.270148 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:07.751720 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.751682 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tf742"] Jul 10 17:52:07.755381 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.755361 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.757686 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.757656 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zxrkb\"" Jul 10 17:52:07.757794 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.757764 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 17:52:07.757849 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.757764 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 17:52:07.757904 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.757856 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 17:52:07.763112 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.763088 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tf742"] Jul 10 17:52:07.776022 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.775969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vdm\" (UniqueName: \"kubernetes.io/projected/9bc7a468-7b19-4195-a042-2fbaaa1cf095-kube-api-access-78vdm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.776145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.776070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.776145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.776093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.776145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.776130 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.776332 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.776196 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.776332 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.776218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.777184 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.777162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:07.877195 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877195 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vdm\" (UniqueName: \"kubernetes.io/projected/9bc7a468-7b19-4195-a042-2fbaaa1cf095-kube-api-access-78vdm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877321 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877385 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877349 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877507 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877409 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877742 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877821 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.877821 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.877771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.879622 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.879602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bc7a468-7b19-4195-a042-2fbaaa1cf095-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.879772 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.879756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bc7a468-7b19-4195-a042-2fbaaa1cf095-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:07.885433 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:07.885411 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vdm\" (UniqueName: \"kubernetes.io/projected/9bc7a468-7b19-4195-a042-2fbaaa1cf095-kube-api-access-78vdm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tf742\" (UID: \"9bc7a468-7b19-4195-a042-2fbaaa1cf095\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:08.067834 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:08.067752 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:08.198087 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:08.198059 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tf742"] Jul 10 17:52:08.200568 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:52:08.200517 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc7a468_7b19_4195_a042_2fbaaa1cf095.slice/crio-a8c9b1765ecb5e3c50d0c56eec598ae3999f2f8f380626ce8aeba11007c5128d WatchSource:0}: Error finding container a8c9b1765ecb5e3c50d0c56eec598ae3999f2f8f380626ce8aeba11007c5128d: Status 404 returned error can't find the container with id a8c9b1765ecb5e3c50d0c56eec598ae3999f2f8f380626ce8aeba11007c5128d Jul 10 17:52:08.545145 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:08.545096 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" event={"ID":"9bc7a468-7b19-4195-a042-2fbaaa1cf095","Type":"ContainerStarted","Data":"a8c9b1765ecb5e3c50d0c56eec598ae3999f2f8f380626ce8aeba11007c5128d"} Jul 10 17:52:09.551457 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:09.551352 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" event={"ID":"9bc7a468-7b19-4195-a042-2fbaaa1cf095","Type":"ContainerStarted","Data":"8d4c1bde39fe356b9ee04703e9c604dc6ee93423b31b6418624df0e4b4f5be4a"} Jul 10 17:52:09.571096 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:09.571038 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" podStartSLOduration=1.4914246119999999 podStartE2EDuration="2.571021088s" totalImagesPullingTime="1.079596476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:52:09.569214005 +0000 UTC m=+796.630029345" watchObservedRunningTime="2026-07-10 17:52:09.571021088 +0000 UTC m=+796.631836422" Jul 10 17:52:16.271777 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:16.271681 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:18.067972 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:18.067937 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:18.068388 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:18.068036 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:18.080327 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:18.080297 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:18.604959 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:18.604933 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tf742" Jul 10 17:52:23.279305 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:23.279272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:27.746138 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.746106 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v"] Jul 10 17:52:27.749346 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.749326 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.751700 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.751681 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 17:52:27.758578 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.758554 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v"] Jul 10 17:52:27.783104 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.783076 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:27.858254 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858223 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.858382 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858273 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.858382 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.858382 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.858485 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858437 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e220bf78-46b6-4c28-8340-74b51adfecfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.858485 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.858460 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbn72\" (UniqueName: \"kubernetes.io/projected/e220bf78-46b6-4c28-8340-74b51adfecfa-kube-api-access-pbn72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959607 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959769 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959635 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959769 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959769 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959927 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e220bf78-46b6-4c28-8340-74b51adfecfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.959927 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.959803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbn72\" (UniqueName: \"kubernetes.io/projected/e220bf78-46b6-4c28-8340-74b51adfecfa-kube-api-access-pbn72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.960114 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.960087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.960225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.960107 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.960225 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.960164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.961754 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.961732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e220bf78-46b6-4c28-8340-74b51adfecfa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.962500 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.962479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e220bf78-46b6-4c28-8340-74b51adfecfa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:27.967696 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:27.967673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbn72\" (UniqueName: \"kubernetes.io/projected/e220bf78-46b6-4c28-8340-74b51adfecfa-kube-api-access-pbn72\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jdd8v\" (UID: \"e220bf78-46b6-4c28-8340-74b51adfecfa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:28.060719 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:28.060656 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:28.202979 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:28.202942 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v"] Jul 10 17:52:28.204386 ip-10-0-133-236 kubenswrapper[2582]: W0710 17:52:28.204360 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode220bf78_46b6_4c28_8340_74b51adfecfa.slice/crio-23b34c71892a1a3829e98afb2c3d311d60b7389f4d23d80719293750edc11f21 WatchSource:0}: Error finding container 23b34c71892a1a3829e98afb2c3d311d60b7389f4d23d80719293750edc11f21: Status 404 returned error can't find the container with id 23b34c71892a1a3829e98afb2c3d311d60b7389f4d23d80719293750edc11f21 Jul 10 17:52:28.638238 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:28.638165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" event={"ID":"e220bf78-46b6-4c28-8340-74b51adfecfa","Type":"ContainerStarted","Data":"3c3595c4fcbd14a14847129a73fea9314352e0cb5a80a8523758a0f975cfb50d"} Jul 10 17:52:28.638238 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:28.638210 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" event={"ID":"e220bf78-46b6-4c28-8340-74b51adfecfa","Type":"ContainerStarted","Data":"23b34c71892a1a3829e98afb2c3d311d60b7389f4d23d80719293750edc11f21"} Jul 10 17:52:28.657008 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:28.656953 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" podStartSLOduration=1.485920127 podStartE2EDuration="1.656938149s" totalImagesPullingTime="171.018022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:52:28.655043099 +0000 UTC m=+815.715858431" watchObservedRunningTime="2026-07-10 17:52:28.656938149 +0000 UTC m=+815.717753484" Jul 10 17:52:31.790454 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:31.790417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:52:38.060771 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:38.060736 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:38.061181 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:38.060834 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:38.073282 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:38.073261 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:52:38.697415 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:52:38.697385 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jdd8v" Jul 10 17:53:28.592441 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:28.592407 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:28.662044 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:28.662012 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:29.733943 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:29.733908 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:29.837495 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:29.837439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:53:30.011146 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.011052 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:53:30.011348 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.011317 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" podUID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" containerName="authorino" containerID="cri-o://08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd" gracePeriod=30 Jul 10 17:53:30.268947 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.268892 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:53:30.404085 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.404050 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djphw\" (UniqueName: \"kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw\") pod \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " Jul 10 17:53:30.404277 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.404165 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert\") pod \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\" (UID: \"cfff7908-36e4-4a1d-bacd-b5303e4c4154\") " Jul 10 17:53:30.406377 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.406349 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw" (OuterVolumeSpecName: "kube-api-access-djphw") pod "cfff7908-36e4-4a1d-bacd-b5303e4c4154" (UID: "cfff7908-36e4-4a1d-bacd-b5303e4c4154"). InnerVolumeSpecName "kube-api-access-djphw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:53:30.416060 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.416031 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cfff7908-36e4-4a1d-bacd-b5303e4c4154" (UID: "cfff7908-36e4-4a1d-bacd-b5303e4c4154"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:53:30.505489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.505464 2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cfff7908-36e4-4a1d-bacd-b5303e4c4154-tls-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:53:30.505489 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.505487 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djphw\" (UniqueName: \"kubernetes.io/projected/cfff7908-36e4-4a1d-bacd-b5303e4c4154-kube-api-access-djphw\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:53:30.929128 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.929094 2582 generic.go:356] "Generic (PLEG): container finished" podID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" containerID="08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd" exitCode=0 Jul 10 17:53:30.929558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.929142 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" Jul 10 17:53:30.929558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.929162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" event={"ID":"cfff7908-36e4-4a1d-bacd-b5303e4c4154","Type":"ContainerDied","Data":"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd"} Jul 10 17:53:30.929558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.929194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f76fc5985-mpzk9" event={"ID":"cfff7908-36e4-4a1d-bacd-b5303e4c4154","Type":"ContainerDied","Data":"6f8105a164ebf05095088a8bbeb525a3f4addfcedf018c76a64c89f76755c6a2"} Jul 10 17:53:30.929558 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.929209 2582 scope.go:117] "RemoveContainer" containerID="08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd" Jul 10 17:53:30.938448 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.938432 2582 scope.go:117] "RemoveContainer" containerID="08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd" Jul 10 17:53:30.938695 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:53:30.938677 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd\": container with ID starting with 08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd not found: ID does not exist" containerID="08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd" Jul 10 17:53:30.938743 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.938705 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd"} err="failed to get container status \"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd\": rpc error: code = NotFound desc = could not find container \"08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd\": container with ID starting with 08a30b46dbf4723a938588984cf9e4254f3d040fb397c001f6665d8a78a6c1bd not found: ID does not exist" Jul 10 17:53:30.956950 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.956927 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:53:30.962801 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:30.962777 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f76fc5985-mpzk9"] Jul 10 17:53:31.432394 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:31.432360 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" path="/var/lib/kubelet/pods/cfff7908-36e4-4a1d-bacd-b5303e4c4154/volumes" Jul 10 17:53:53.367300 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:53.367226 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:53:53.369045 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:53:53.369020 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:54:33.306337 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:54:33.306268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:34.428410 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:54:34.428375 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:39.832623 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:54:39.832584 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:41.602910 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:54:41.602873 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:54:48.823521 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:54:48.823478 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:25.280590 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:25.280498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:25.376417 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:25.376387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:25.500315 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:25.500280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:31.418370 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:31.418314 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:31.534310 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:31.534272 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:31.622873 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:31.622843 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:31.778168 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:31.778080 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:37.650269 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:37.650228 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:55:37.749857 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:55:37.749823 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:56:01.195366 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:56:01.195328 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:57:26.247183 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:57:26.247155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:57:29.516968 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:57:29.516930 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:57:48.317047 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:57:48.317013 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:57:48.370804 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:57:48.370775 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:58:19.973114 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:58:19.973032 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:58:53.395345 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:58:53.395316 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:58:53.397799 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:58:53.397776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 17:59:34.375229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:34.375185 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:59:34.375735 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:34.375403 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerName="manager" containerID="cri-o://d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af" gracePeriod=10 Jul 10 17:59:37.944015 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:37.943970 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:59:38.013232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.013157 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume\") pod \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " Jul 10 17:59:38.013232 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.013216 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnflb\" (UniqueName: \"kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb\") pod \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\" (UID: \"7449cdf1-b8e1-4539-82b7-0b2d256c383d\") " Jul 10 17:59:38.013512 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.013490 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7449cdf1-b8e1-4539-82b7-0b2d256c383d" (UID: "7449cdf1-b8e1-4539-82b7-0b2d256c383d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 17:59:38.015299 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.015270 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb" (OuterVolumeSpecName: "kube-api-access-cnflb") pod "7449cdf1-b8e1-4539-82b7-0b2d256c383d" (UID: "7449cdf1-b8e1-4539-82b7-0b2d256c383d"). InnerVolumeSpecName "kube-api-access-cnflb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:59:38.114016 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.113964 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7449cdf1-b8e1-4539-82b7-0b2d256c383d-extensions-socket-volume\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:59:38.114016 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.114011 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnflb\" (UniqueName: \"kubernetes.io/projected/7449cdf1-b8e1-4539-82b7-0b2d256c383d-kube-api-access-cnflb\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 17:59:38.654818 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.654781 2582 generic.go:356] "Generic (PLEG): container finished" podID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerID="d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af" exitCode=0 Jul 10 17:59:38.654985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.654823 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" Jul 10 17:59:38.654985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.654868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" event={"ID":"7449cdf1-b8e1-4539-82b7-0b2d256c383d","Type":"ContainerDied","Data":"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af"} Jul 10 17:59:38.654985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.654920 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" event={"ID":"7449cdf1-b8e1-4539-82b7-0b2d256c383d","Type":"ContainerDied","Data":"db122e6577e00e5d1bb0d138a158b011b9a392f31066b0dd45387e75b43caa22"} Jul 10 17:59:38.654985 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.654942 2582 scope.go:117] "RemoveContainer" containerID="d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af" Jul 10 17:59:38.664703 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.664689 2582 scope.go:117] "RemoveContainer" containerID="d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af" Jul 10 17:59:38.664955 ip-10-0-133-236 kubenswrapper[2582]: E0710 17:59:38.664934 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af\": container with ID starting with d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af not found: ID does not exist" containerID="d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af" Jul 10 17:59:38.665017 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.664969 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af"} err="failed to get container status \"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af\": rpc error: code = NotFound desc = could not find container \"d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af\": container with ID starting with d7644814570d48f0e4935e6550daba61525538b5305ebd5aa048abbb9d4870af not found: ID does not exist" Jul 10 17:59:38.676463 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.676437 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:59:38.680707 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.680686 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v"] Jul 10 17:59:38.927229 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:38.927142 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zbk7v" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 17:59:39.432647 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:39.432619 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" path="/var/lib/kubelet/pods/7449cdf1-b8e1-4539-82b7-0b2d256c383d/volumes" Jul 10 17:59:49.567697 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:49.567666 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 17:59:50.807044 ip-10-0-133-236 kubenswrapper[2582]: I0710 17:59:50.807009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:00:00.145036 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145002 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145338 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerName="manager" Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145348 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerName="manager" Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145358 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" containerName="authorino" Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145363 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" containerName="authorino" Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145448 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfff7908-36e4-4a1d-bacd-b5303e4c4154" containerName="authorino" Jul 10 18:00:00.145501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.145458 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7449cdf1-b8e1-4539-82b7-0b2d256c383d" containerName="manager" Jul 10 18:00:00.149306 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.149291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" Jul 10 18:00:00.151907 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.151883 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jxczz\"" Jul 10 18:00:00.157962 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.157936 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:00:00.160449 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.160430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" Jul 10 18:00:00.293107 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.293079 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:00:00.300378 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.300354 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:00:00.755929 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:00.755895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" event={"ID":"6090ef6f-8245-43c2-b6e2-3f3b1a3803cc","Type":"ContainerStarted","Data":"65463f5263088d3cea00cfec2c75d123576de75cb69229881ddb33a05c104f91"} Jul 10 18:00:03.773263 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:03.773227 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" event={"ID":"6090ef6f-8245-43c2-b6e2-3f3b1a3803cc","Type":"ContainerStarted","Data":"0be30b3de4d561dac5bec6c6f83d15ab1ff68af2ac781b04eca162c279602557"} Jul 10 18:00:03.788880 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:03.788835 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" podStartSLOduration=1.043519506 podStartE2EDuration="3.788819401s" totalImagesPullingTime="2.745299895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:03.787235194 +0000 UTC m=+1270.848050530" watchObservedRunningTime="2026-07-10 18:00:03.788819401 +0000 UTC m=+1270.849634783" Jul 10 18:00:35.537805 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.537771 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh"] Jul 10 18:00:35.541384 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.541367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.543884 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.543865 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5zmsz\"" Jul 10 18:00:35.553425 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.553403 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh"] Jul 10 18:00:35.614443 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.614419 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a50536-0a7f-42c3-8237-a5ebeb60cda1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.614579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.614462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbmtw\" (UniqueName: \"kubernetes.io/projected/96a50536-0a7f-42c3-8237-a5ebeb60cda1-kube-api-access-wbmtw\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.715794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.715762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a50536-0a7f-42c3-8237-a5ebeb60cda1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.715961 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.715811 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbmtw\" (UniqueName: \"kubernetes.io/projected/96a50536-0a7f-42c3-8237-a5ebeb60cda1-kube-api-access-wbmtw\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.716284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.716238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/96a50536-0a7f-42c3-8237-a5ebeb60cda1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.725407 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.725380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbmtw\" (UniqueName: \"kubernetes.io/projected/96a50536-0a7f-42c3-8237-a5ebeb60cda1-kube-api-access-wbmtw\") pod \"kuadrant-operator-controller-manager-86878c769f-ngsmh\" (UID: \"96a50536-0a7f-42c3-8237-a5ebeb60cda1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:35.851524 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:35.851436 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:36.004112 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:36.004090 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh"] Jul 10 18:00:36.006289 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:00:36.006251 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a50536_0a7f_42c3_8237_a5ebeb60cda1.slice/crio-46cdbc3fc887e2f18b5247f64559a4c897d0651bffcfdc0b00e282817f53c1ca WatchSource:0}: Error finding container 46cdbc3fc887e2f18b5247f64559a4c897d0651bffcfdc0b00e282817f53c1ca: Status 404 returned error can't find the container with id 46cdbc3fc887e2f18b5247f64559a4c897d0651bffcfdc0b00e282817f53c1ca Jul 10 18:00:36.927110 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:36.927080 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" event={"ID":"96a50536-0a7f-42c3-8237-a5ebeb60cda1","Type":"ContainerStarted","Data":"02bdd968eb4126505e6f871b3632c4ce0f690b992f7dc91f68f1c8325287c78f"} Jul 10 18:00:36.927492 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:36.927130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" event={"ID":"96a50536-0a7f-42c3-8237-a5ebeb60cda1","Type":"ContainerStarted","Data":"46cdbc3fc887e2f18b5247f64559a4c897d0651bffcfdc0b00e282817f53c1ca"} Jul 10 18:00:36.927492 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:36.927181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:00:36.948479 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:36.948430 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" podStartSLOduration=1.948415577 podStartE2EDuration="1.948415577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:00:36.94667597 +0000 UTC m=+1304.007491305" watchObservedRunningTime="2026-07-10 18:00:36.948415577 +0000 UTC m=+1304.009230912" Jul 10 18:00:47.939655 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:00:47.939624 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ngsmh" Jul 10 18:01:00.010351 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:00.010314 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:01:00.010820 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:00.010518 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" podUID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" containerName="cleanup" containerID="cri-o://0be30b3de4d561dac5bec6c6f83d15ab1ff68af2ac781b04eca162c279602557" gracePeriod=30 Jul 10 18:01:30.180593 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:30.180557 2582 generic.go:356] "Generic (PLEG): container finished" podID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" containerID="0be30b3de4d561dac5bec6c6f83d15ab1ff68af2ac781b04eca162c279602557" exitCode=137 Jul 10 18:01:30.180972 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:30.180656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" event={"ID":"6090ef6f-8245-43c2-b6e2-3f3b1a3803cc","Type":"ContainerDied","Data":"0be30b3de4d561dac5bec6c6f83d15ab1ff68af2ac781b04eca162c279602557"} Jul 10 18:01:30.180972 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:30.180729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" event={"ID":"6090ef6f-8245-43c2-b6e2-3f3b1a3803cc","Type":"ContainerDied","Data":"65463f5263088d3cea00cfec2c75d123576de75cb69229881ddb33a05c104f91"} Jul 10 18:01:30.180972 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:30.180754 2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65463f5263088d3cea00cfec2c75d123576de75cb69229881ddb33a05c104f91" Jul 10 18:01:30.192958 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:30.192942 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" Jul 10 18:01:31.185142 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:31.185111 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b" Jul 10 18:01:31.220185 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:31.220156 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:01:31.231149 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:31.231123 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728440-mxn2b"] Jul 10 18:01:31.432537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:01:31.432496 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" path="/var/lib/kubelet/pods/6090ef6f-8245-43c2-b6e2-3f3b1a3803cc/volumes" Jul 10 18:03:53.425588 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:03:53.425560 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:03:53.428261 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:03:53.428239 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:05:36.630525 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:05:36.630487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:05:41.523318 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:05:41.523286 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:05:41.578827 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:05:41.578801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:17.377468 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:17.377432 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:21.660614 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:21.660582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:31.349741 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:31.349697 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:39.617366 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:39.617330 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:44.756883 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:44.756847 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:47.761824 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:47.761787 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:47.861769 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:47.861739 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:48.241364 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:48.241331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:53.501842 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:53.501810 2582 scope.go:117] "RemoveContainer" containerID="0be30b3de4d561dac5bec6c6f83d15ab1ff68af2ac781b04eca162c279602557" Jul 10 18:06:54.270928 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.270892 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:54.374486 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.374450 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:54.424075 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.424045 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:06:54.424398 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.424386 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" containerName="cleanup" Jul 10 18:06:54.424446 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.424400 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" containerName="cleanup" Jul 10 18:06:54.424484 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.424476 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6090ef6f-8245-43c2-b6e2-3f3b1a3803cc" containerName="cleanup" Jul 10 18:06:54.426394 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.426372 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.428761 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.428743 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-29a83520-openshift-default-dockercfg-qt2br\"" Jul 10 18:06:54.450920 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.450898 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:06:54.460759 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.460735 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:54.519456 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519562 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519645 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fwt\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519701 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.519890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.519882 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.544960 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.544886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:54.620999 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.620969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fwt\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621187 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621522 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621522 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621522 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621667 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621599 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621667 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621939 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621716 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.621939 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.621918 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.623408 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.623389 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.623762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.623747 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.632029 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.631975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fwt\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.632237 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.632220 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token\") pod \"e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.737137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.737108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:54.879224 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.879200 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:06:54.881135 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:06:54.881108 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7919d35_e89c_44fe_ae0e_3e112b26f32a.slice/crio-733ab7e89af35eb4bc93089c6848a5e96bdb078de70bdbde3b9e0d471d911066 WatchSource:0}: Error finding container 733ab7e89af35eb4bc93089c6848a5e96bdb078de70bdbde3b9e0d471d911066: Status 404 returned error can't find the container with id 733ab7e89af35eb4bc93089c6848a5e96bdb078de70bdbde3b9e0d471d911066 Jul 10 18:06:54.883330 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.883295 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:06:54.883434 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.883361 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:06:54.883434 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:54.883386 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:06:55.635346 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.635298 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:55.716776 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.716739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" event={"ID":"b7919d35-e89c-44fe-ae0e-3e112b26f32a","Type":"ContainerStarted","Data":"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422"} Jul 10 18:06:55.716944 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.716803 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" event={"ID":"b7919d35-e89c-44fe-ae0e-3e112b26f32a","Type":"ContainerStarted","Data":"733ab7e89af35eb4bc93089c6848a5e96bdb078de70bdbde3b9e0d471d911066"} Jul 10 18:06:55.737688 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.737658 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:55.739484 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.739456 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:55.742662 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.742641 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:55.749594 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.749554 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" podStartSLOduration=1.749541052 podStartE2EDuration="1.749541052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:06:55.746533453 +0000 UTC m=+1682.807348789" watchObservedRunningTime="2026-07-10 18:06:55.749541052 +0000 UTC m=+1682.810356386" Jul 10 18:06:55.843184 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:55.843150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:56.460736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:56.460691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:56.559674 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:56.559627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:06:56.722303 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:56.722212 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:56.723155 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:56.723132 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:06:58.429427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:06:58.429399 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:00.752231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:00.752202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:00.926888 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:00.926844 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:07:00.927155 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:00.927120 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" podUID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" containerName="istio-proxy" containerID="cri-o://9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422" gracePeriod=30 Jul 10 18:07:00.951073 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:00.951033 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:01.056822 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:01.056727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:01.214697 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:01.214668 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:02.678037 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:02.678007 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:06.838726 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:06.838660 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:06.939772 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:06.939743 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:07.055953 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.055921 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:07.058893 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.058870 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.062296 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.062268 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a73b4da0-openshift-default-dockercfg-f8r5k\"" Jul 10 18:07:07.075252 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.075229 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:07.101764 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.101678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:07.148382 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148385 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnr2r\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148613 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148651 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148651 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148737 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148657 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148737 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148688 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.148817 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.148740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.198372 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.198304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:07.250036 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.249977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250058 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250121 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250437 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250437 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250437 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnr2r\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250437 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250641 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250469 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250641 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250798 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.250908 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.250889 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.251047 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.251030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.251119 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.251093 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.252759 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.252738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.252858 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.252757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.261230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.261193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnr2r\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.261380 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.261360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token\") pod \"e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.282695 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.282674 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:07:07.351330 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351299 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351508 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351355 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351508 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351485 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351639 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351566 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351639 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351593 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351639 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351628 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351821 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351821 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351720 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9fwt\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.351821 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.351755 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs\") pod \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\" (UID: \"b7919d35-e89c-44fe-ae0e-3e112b26f32a\") " Jul 10 18:07:07.353289 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.353266 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:07:07.355350 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.355295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:07.355350 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.355312 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:07.355526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.355388 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:07.356199 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.356172 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:07.357263 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.357238 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:07.357363 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.357241 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt" (OuterVolumeSpecName: "kube-api-access-d9fwt") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "kube-api-access-d9fwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:07.357363 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.357314 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:07:07.357451 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.357428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7919d35-e89c-44fe-ae0e-3e112b26f32a" (UID: "b7919d35-e89c-44fe-ae0e-3e112b26f32a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:07.413747 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.413712 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:07.453638 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453609 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453644 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453660 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453677 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453688 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453700 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453711 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9fwt\" (UniqueName: \"kubernetes.io/projected/b7919d35-e89c-44fe-ae0e-3e112b26f32a-kube-api-access-d9fwt\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453724 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7919d35-e89c-44fe-ae0e-3e112b26f32a-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.453782 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.453737 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7919d35-e89c-44fe-ae0e-3e112b26f32a-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:07.555653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.555622 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:07.556913 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:07:07.556890 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999fd015_47b9_4611_875e_b64cae29fb26.slice/crio-91d7f70dbe69b5419cb20404c6776ebb451cb758d9dbec78baf2a6965fb4869a WatchSource:0}: Error finding container 91d7f70dbe69b5419cb20404c6776ebb451cb758d9dbec78baf2a6965fb4869a: Status 404 returned error can't find the container with id 91d7f70dbe69b5419cb20404c6776ebb451cb758d9dbec78baf2a6965fb4869a Jul 10 18:07:07.558963 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.558933 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:07.559064 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.559047 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:07.559112 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.559084 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:07.775628 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.775589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" event={"ID":"999fd015-47b9-4611-875e-b64cae29fb26","Type":"ContainerStarted","Data":"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4"} Jul 10 18:07:07.775803 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.775655 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" event={"ID":"999fd015-47b9-4611-875e-b64cae29fb26","Type":"ContainerStarted","Data":"91d7f70dbe69b5419cb20404c6776ebb451cb758d9dbec78baf2a6965fb4869a"} Jul 10 18:07:07.777062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.777023 2582 generic.go:356] "Generic (PLEG): container finished" podID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" containerID="9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422" exitCode=0 Jul 10 18:07:07.777162 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.777115 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" Jul 10 18:07:07.777162 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.777127 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" event={"ID":"b7919d35-e89c-44fe-ae0e-3e112b26f32a","Type":"ContainerDied","Data":"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422"} Jul 10 18:07:07.777228 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.777177 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg" event={"ID":"b7919d35-e89c-44fe-ae0e-3e112b26f32a","Type":"ContainerDied","Data":"733ab7e89af35eb4bc93089c6848a5e96bdb078de70bdbde3b9e0d471d911066"} Jul 10 18:07:07.777228 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.777208 2582 scope.go:117] "RemoveContainer" containerID="9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422" Jul 10 18:07:07.786343 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.786321 2582 scope.go:117] "RemoveContainer" containerID="9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422" Jul 10 18:07:07.786616 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:07:07.786591 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422\": container with ID starting with 9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422 not found: ID does not exist" containerID="9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422" Jul 10 18:07:07.786695 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.786628 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422"} err="failed to get container status \"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422\": rpc error: code = NotFound desc = could not find container \"9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422\": container with ID starting with 9bbf2d36918836762eed47849e38ab417af8475b9685e9ee08bc5b2f79bca422 not found: ID does not exist" Jul 10 18:07:07.800092 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.800026 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" podStartSLOduration=0.800009171 podStartE2EDuration="800.009171ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:07.799810875 +0000 UTC m=+1694.860626224" watchObservedRunningTime="2026-07-10 18:07:07.800009171 +0000 UTC m=+1694.860824496" Jul 10 18:07:07.813759 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.813733 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:07:07.819096 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:07.819076 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-29a83520-openshift-default-7cdb69d9dc-75dsg"] Jul 10 18:07:08.325423 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.325367 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:08.414251 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.414219 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:08.419387 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.419366 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:08.540382 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.540345 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:08.702502 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.702472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:08.783111 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.783086 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:08.783941 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.783921 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:08.898332 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:08.898287 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:09.344716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:09.344683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:09.433259 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:09.433226 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" path="/var/lib/kubelet/pods/b7919d35-e89c-44fe-ae0e-3e112b26f32a/volumes" Jul 10 18:07:09.449188 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:09.449162 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:10.444512 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:10.444440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:13.384295 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:13.384266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:13.859410 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:13.859373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:19.503555 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.503521 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:19.504073 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.503769 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" podUID="999fd015-47b9-4611-875e-b64cae29fb26" containerName="istio-proxy" containerID="cri-o://427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4" gracePeriod=30 Jul 10 18:07:19.546935 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.546904 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:19.645287 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.645241 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:19.650565 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.650545 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:19.774501 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774535 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774560 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnr2r\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774581 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774978 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774978 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774756 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket\") pod \"999fd015-47b9-4611-875e-b64cae29fb26\" (UID: \"999fd015-47b9-4611-875e-b64cae29fb26\") " Jul 10 18:07:19.774978 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.774842 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:19.775172 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.775138 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.775247 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.775221 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:19.775322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.775295 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:19.775525 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.775500 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data" (OuterVolumeSpecName: "istio-data") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:19.775641 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.775620 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:07:19.776899 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.776874 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r" (OuterVolumeSpecName: "kube-api-access-dnr2r") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "kube-api-access-dnr2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:19.776899 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.776888 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token" (OuterVolumeSpecName: "istio-token") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:19.777334 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.777309 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:19.777429 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.777325 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "999fd015-47b9-4611-875e-b64cae29fb26" (UID: "999fd015-47b9-4611-875e-b64cae29fb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:07:19.846383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.846346 2582 generic.go:356] "Generic (PLEG): container finished" podID="999fd015-47b9-4611-875e-b64cae29fb26" containerID="427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4" exitCode=0 Jul 10 18:07:19.846526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.846398 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" Jul 10 18:07:19.846526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.846439 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" event={"ID":"999fd015-47b9-4611-875e-b64cae29fb26","Type":"ContainerDied","Data":"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4"} Jul 10 18:07:19.846526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.846497 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq" event={"ID":"999fd015-47b9-4611-875e-b64cae29fb26","Type":"ContainerDied","Data":"91d7f70dbe69b5419cb20404c6776ebb451cb758d9dbec78baf2a6965fb4869a"} Jul 10 18:07:19.846526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.846521 2582 scope.go:117] "RemoveContainer" containerID="427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4" Jul 10 18:07:19.855718 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.855703 2582 scope.go:117] "RemoveContainer" containerID="427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4" Jul 10 18:07:19.855968 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:07:19.855950 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4\": container with ID starting with 427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4 not found: ID does not exist" containerID="427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4" Jul 10 18:07:19.856094 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.855975 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4"} err="failed to get container status \"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4\": rpc error: code = NotFound desc = could not find container \"427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4\": container with ID starting with 427b07861946b89c287b738fb5ef5b9383f8cd267aa5985602689d46b7766de4 not found: ID does not exist" Jul 10 18:07:19.867691 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.867671 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:19.873206 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.873185 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a73b4da0-openshift-default-7d698655dc-cjccq"] Jul 10 18:07:19.875686 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875670 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnr2r\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-kube-api-access-dnr2r\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875690 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875702 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999fd015-47b9-4611-875e-b64cae29fb26-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875710 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875719 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999fd015-47b9-4611-875e-b64cae29fb26-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875727 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875735 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999fd015-47b9-4611-875e-b64cae29fb26-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:19.875762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:19.875743 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999fd015-47b9-4611-875e-b64cae29fb26-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:21.432555 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:21.432523 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="999fd015-47b9-4611-875e-b64cae29fb26" path="/var/lib/kubelet/pods/999fd015-47b9-4611-875e-b64cae29fb26/volumes" Jul 10 18:07:26.271599 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.271564 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:26.373985 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.373951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:26.489218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489185 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:26.489565 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489551 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="999fd015-47b9-4611-875e-b64cae29fb26" containerName="istio-proxy" Jul 10 18:07:26.489565 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489566 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="999fd015-47b9-4611-875e-b64cae29fb26" containerName="istio-proxy" Jul 10 18:07:26.489716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489589 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" containerName="istio-proxy" Jul 10 18:07:26.489716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489594 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" containerName="istio-proxy" Jul 10 18:07:26.489716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489644 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7919d35-e89c-44fe-ae0e-3e112b26f32a" containerName="istio-proxy" Jul 10 18:07:26.489716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.489654 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="999fd015-47b9-4611-875e-b64cae29fb26" containerName="istio-proxy" Jul 10 18:07:26.491741 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.491717 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.494537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.494508 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7bc3b5cb-openshift-default-dockercfg-b5n9g\"" Jul 10 18:07:26.510650 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.510626 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:26.542042 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.541939 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:26.639403 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639369 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzl6v\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639576 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.639736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.639649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740507 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740693 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740525 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740693 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740693 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740872 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzl6v\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740872 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.740872 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740836 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740910 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.740915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741233 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.741209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741340 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.741315 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.741390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.741660 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.741639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.743041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.743023 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.743393 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.743374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.749198 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.749179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.751684 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.751664 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzl6v\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v\") pod \"e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.804591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.804477 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:26.870617 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.870566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:26.978486 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.978462 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:26.981695 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:07:26.981654 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffba06d_53bb_4308_b33d_24c173c90750.slice/crio-3b3e3ed14d22af0a1d886d70ac6439b84d51ba41b0f7d33b1f98bac721be9b6f WatchSource:0}: Error finding container 3b3e3ed14d22af0a1d886d70ac6439b84d51ba41b0f7d33b1f98bac721be9b6f: Status 404 returned error can't find the container with id 3b3e3ed14d22af0a1d886d70ac6439b84d51ba41b0f7d33b1f98bac721be9b6f Jul 10 18:07:26.984200 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.984167 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:26.984313 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.984237 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:26.984313 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:26.984270 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:07:27.025209 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.025179 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:27.782014 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.781958 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:27.889504 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.889476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" event={"ID":"1ffba06d-53bb-4308-b33d-24c173c90750","Type":"ContainerStarted","Data":"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b"} Jul 10 18:07:27.889671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.889526 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" event={"ID":"1ffba06d-53bb-4308-b33d-24c173c90750","Type":"ContainerStarted","Data":"3b3e3ed14d22af0a1d886d70ac6439b84d51ba41b0f7d33b1f98bac721be9b6f"} Jul 10 18:07:27.955568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.955518 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" podStartSLOduration=1.955502375 podStartE2EDuration="1.955502375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:07:27.954645532 +0000 UTC m=+1715.015460870" watchObservedRunningTime="2026-07-10 18:07:27.955502375 +0000 UTC m=+1715.016317762" Jul 10 18:07:27.966217 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:27.966189 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:28.088615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.088530 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:28.662984 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.662951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:28.760137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.760109 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:28.805295 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.805260 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:28.810134 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.810111 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:28.894638 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.894604 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:28.895437 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:28.895418 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:30.310478 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:30.310446 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:32.224682 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:32.224649 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:32.426845 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:32.426815 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:32.427061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:32.426978 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" podUID="1ffba06d-53bb-4308-b33d-24c173c90750" containerName="istio-proxy" containerID="cri-o://d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b" gracePeriod=30 Jul 10 18:07:32.466412 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:32.466385 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:32.603048 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:32.602926 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:33.096674 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:33.096617 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:34.452530 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:34.452500 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:37.677867 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.677843 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:37.744176 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744095 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744176 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744150 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744191 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744228 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744253 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzl6v\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744362 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744383 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs\") pod \"1ffba06d-53bb-4308-b33d-24c173c90750\" (UID: \"1ffba06d-53bb-4308-b33d-24c173c90750\") " Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744456 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744604 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744664 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744727 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ffba06d-53bb-4308-b33d-24c173c90750-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.744762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744742 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.745054 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744779 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.745054 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744794 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.745054 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.744797 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:37.746755 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.746727 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v" (OuterVolumeSpecName: "kube-api-access-qzl6v") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "kube-api-access-qzl6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:37.746755 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.746735 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:07:37.746944 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.746926 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:07:37.747043 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.747020 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ffba06d-53bb-4308-b33d-24c173c90750" (UID: "1ffba06d-53bb-4308-b33d-24c173c90750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:07:37.845658 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.845624 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.845658 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.845654 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzl6v\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-kube-api-access-qzl6v\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.845658 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.845667 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ffba06d-53bb-4308-b33d-24c173c90750-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.845894 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.845678 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ffba06d-53bb-4308-b33d-24c173c90750-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.845894 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.845689 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ffba06d-53bb-4308-b33d-24c173c90750-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:07:37.938905 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.938865 2582 generic.go:356] "Generic (PLEG): container finished" podID="1ffba06d-53bb-4308-b33d-24c173c90750" containerID="d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b" exitCode=0 Jul 10 18:07:37.938905 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.938905 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" Jul 10 18:07:37.939153 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.938948 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" event={"ID":"1ffba06d-53bb-4308-b33d-24c173c90750","Type":"ContainerDied","Data":"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b"} Jul 10 18:07:37.939153 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.939029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67" event={"ID":"1ffba06d-53bb-4308-b33d-24c173c90750","Type":"ContainerDied","Data":"3b3e3ed14d22af0a1d886d70ac6439b84d51ba41b0f7d33b1f98bac721be9b6f"} Jul 10 18:07:37.939153 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.939054 2582 scope.go:117] "RemoveContainer" containerID="d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b" Jul 10 18:07:37.948409 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.948393 2582 scope.go:117] "RemoveContainer" containerID="d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b" Jul 10 18:07:37.948670 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:07:37.948651 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b\": container with ID starting with d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b not found: ID does not exist" containerID="d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b" Jul 10 18:07:37.948737 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.948683 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b"} err="failed to get container status \"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b\": rpc error: code = NotFound desc = could not find container \"d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b\": container with ID starting with d0e20d7cc8cc25e52c1ed6da0ba056c0027cfa3abfcb04799f84bd592b9a296b not found: ID does not exist" Jul 10 18:07:37.962529 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.962501 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:37.968674 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:37.968654 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7bc3b5cb-openshift-default-66bcdb88b7-52j67"] Jul 10 18:07:39.077133 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:39.077102 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:39.185793 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:39.185762 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:39.433008 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:39.432950 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffba06d-53bb-4308-b33d-24c173c90750" path="/var/lib/kubelet/pods/1ffba06d-53bb-4308-b33d-24c173c90750/volumes" Jul 10 18:07:39.706063 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:39.705971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:39.810251 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:39.810220 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:41.329328 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:41.329299 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:47.644149 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:47.644116 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:47.714446 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:47.714418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:47.802008 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:47.801971 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:47.906707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:47.906672 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:48.025201 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:48.025165 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:07:48.413164 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:07:48.413136 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:14.433299 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:14.433265 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:20.506728 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:20.506698 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:20.665672 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:20.665638 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:20.755913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:20.755883 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:29.421763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:29.421727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:29.522415 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:29.522379 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:36.696770 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:36.696736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:42.943757 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:42.943721 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:43.014911 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:43.014874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:43.123411 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:43.123381 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:43.207928 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:43.207852 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:53.455749 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:53.455722 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:08:53.460378 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:53.460357 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:08:53.707446 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:53.707373 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:08:54.608126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:08:54.608100 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:00.989271 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:00.989238 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:01.109750 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:01.109714 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:01.211965 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:01.211925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:01.308980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:01.308896 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:02.176087 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:02.176049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:02.268146 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:02.268115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:02.368955 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:02.368912 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:04.108091 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:04.108057 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:10.127419 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:10.127391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:10.213721 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:10.213688 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:10.822052 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:10.822016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:17.445532 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:17.445494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:17.790711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:17.790609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:17.884641 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:17.884603 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:18.037298 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.037266 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:18.037799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.037778 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffba06d-53bb-4308-b33d-24c173c90750" containerName="istio-proxy" Jul 10 18:09:18.037799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.037798 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffba06d-53bb-4308-b33d-24c173c90750" containerName="istio-proxy" Jul 10 18:09:18.037949 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.037917 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffba06d-53bb-4308-b33d-24c173c90750" containerName="istio-proxy" Jul 10 18:09:18.041086 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.041031 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.043942 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.043921 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f6b351bc-openshift-default-dockercfg-f7lt2\"" Jul 10 18:09:18.070956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.070928 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:18.077101 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.077072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:18.153954 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.153917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154144 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.153960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154144 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154042 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154144 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2qbf\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154301 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154176 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154301 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154301 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154301 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.154491 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.154384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255702 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255660 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255707 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.255904 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2qbf\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.255985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256449 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.256317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256449 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.256405 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256549 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.256465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.256548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.256927 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.256905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.258399 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.258381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.258630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.258608 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.263694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.263675 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2qbf\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.263770 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.263729 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token\") pod \"e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.351353 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.351268 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:18.381381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.381352 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:18.492733 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.492703 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:18.495245 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:09:18.495217 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4858923c_0ae2_4b93_b0f1_782963e800c7.slice/crio-ddcbc2c41cf45024ebeba2b9e4ddbafa25c0b53ce4ecf2bcaabdbe8bf08a67a9 WatchSource:0}: Error finding container ddcbc2c41cf45024ebeba2b9e4ddbafa25c0b53ce4ecf2bcaabdbe8bf08a67a9: Status 404 returned error can't find the container with id ddcbc2c41cf45024ebeba2b9e4ddbafa25c0b53ce4ecf2bcaabdbe8bf08a67a9 Jul 10 18:09:18.497489 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.497456 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:18.497597 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.497523 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:18.497597 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:18.497561 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:19.437310 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:19.437276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerStarted","Data":"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2"} Jul 10 18:09:19.437518 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:19.437320 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerStarted","Data":"ddcbc2c41cf45024ebeba2b9e4ddbafa25c0b53ce4ecf2bcaabdbe8bf08a67a9"} Jul 10 18:09:19.460741 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:19.460689 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podStartSLOduration=1.46067436 podStartE2EDuration="1.46067436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:09:19.459711202 +0000 UTC m=+1826.520526538" watchObservedRunningTime="2026-07-10 18:09:19.46067436 +0000 UTC m=+1826.521489694" Jul 10 18:09:20.351736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:20.351703 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:20.356464 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:20.356440 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:20.441984 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:20.441949 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:20.443104 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:20.443083 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:21.441184 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:21.441150 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:21.532601 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:21.532567 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:21.624951 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:21.624916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:21.722699 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:21.722590 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:22.128373 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:22.128273 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:22.230572 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:22.230533 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:25.940937 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:25.940901 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:26.744347 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:26.744311 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:26.847583 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:26.847549 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:27.144668 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:27.144561 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:33.446857 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:33.446776 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:33.512451 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:33.512406 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:34.085753 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:34.085718 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:34.499121 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:34.498965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:34.603210 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:34.603176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:35.717012 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:35.715918 2582 patch_prober.go:28] interesting pod/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:53006->10.133.0.46:15021: read: connection reset by peer" start-of-body= Jul 10 18:09:35.717012 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:35.716045 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:53006->10.133.0.46:15021: read: connection reset by peer" Jul 10 18:09:36.518625 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.518594 2582 generic.go:356] "Generic (PLEG): container finished" podID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerID="c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2" exitCode=0 Jul 10 18:09:36.518805 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.518645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerDied","Data":"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2"} Jul 10 18:09:36.519080 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.519066 2582 scope.go:117] "RemoveContainer" containerID="c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2" Jul 10 18:09:36.520099 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.520068 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:36.520186 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.520169 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:36.520224 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:36.520215 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:09:37.528464 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:37.528429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerStarted","Data":"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141"} Jul 10 18:09:38.352298 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:38.352266 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:38.352537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:38.352517 2582 patch_prober.go:28] interesting pod/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 18:09:38.352584 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:38.352563 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 18:09:39.351643 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:39.351605 2582 patch_prober.go:28] interesting pod/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 18:09:39.352105 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:39.351671 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 18:09:40.325336 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.325294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:40.351882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.351845 2582 patch_prober.go:28] interesting pod/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 10 18:09:40.352305 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.351913 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 10 18:09:40.427431 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.427387 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:40.589108 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.589020 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:40.589337 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.589304 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" containerID="cri-o://54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141" gracePeriod=30 Jul 10 18:09:40.610040 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.609983 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:40.899460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:40.899353 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:41.004563 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:41.004449 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:41.174017 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:41.173968 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:47.049773 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.049738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:47.154527 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.154494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:47.249478 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.249456 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:47.263949 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.263922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:47.320429 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320342 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320429 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320408 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320470 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320494 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320528 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320561 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2qbf\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320620 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.320653 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320633 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.321062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320641 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert\") pod \"4858923c-0ae2-4b93-b0f1-782963e800c7\" (UID: \"4858923c-0ae2-4b93-b0f1-782963e800c7\") " Jul 10 18:09:47.321062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320788 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.321062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.320978 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:09:47.321391 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.321105 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4858923c-0ae2-4b93-b0f1-782963e800c7-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.321391 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.321120 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.321391 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.321132 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.321391 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.321228 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.321688 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.321640 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.322885 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.322859 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:09:47.322981 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.322883 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:47.322981 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.322894 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf" (OuterVolumeSpecName: "kube-api-access-f2qbf") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "kube-api-access-f2qbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:09:47.323336 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.323314 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4858923c-0ae2-4b93-b0f1-782963e800c7" (UID: "4858923c-0ae2-4b93-b0f1-782963e800c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:09:47.420428 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.420394 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:47.422590 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422567 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.422590 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422593 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.422738 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422605 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.422738 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422618 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4858923c-0ae2-4b93-b0f1-782963e800c7-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.422738 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422630 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2qbf\" (UniqueName: \"kubernetes.io/projected/4858923c-0ae2-4b93-b0f1-782963e800c7-kube-api-access-f2qbf\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.422738 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.422643 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4858923c-0ae2-4b93-b0f1-782963e800c7-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:09:47.577427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.577326 2582 generic.go:356] "Generic (PLEG): container finished" podID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerID="54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141" exitCode=0 Jul 10 18:09:47.577427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.577374 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" Jul 10 18:09:47.577427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.577413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerDied","Data":"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141"} Jul 10 18:09:47.577694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.577468 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2" event={"ID":"4858923c-0ae2-4b93-b0f1-782963e800c7","Type":"ContainerDied","Data":"ddcbc2c41cf45024ebeba2b9e4ddbafa25c0b53ce4ecf2bcaabdbe8bf08a67a9"} Jul 10 18:09:47.577694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.577491 2582 scope.go:117] "RemoveContainer" containerID="54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141" Jul 10 18:09:47.586223 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.586198 2582 scope.go:117] "RemoveContainer" containerID="c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2" Jul 10 18:09:47.594173 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.594150 2582 scope.go:117] "RemoveContainer" containerID="54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141" Jul 10 18:09:47.594428 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:09:47.594410 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141\": container with ID starting with 54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141 not found: ID does not exist" containerID="54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141" Jul 10 18:09:47.594512 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.594442 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141"} err="failed to get container status \"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141\": rpc error: code = NotFound desc = could not find container \"54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141\": container with ID starting with 54d853f50db1ea780f9f67ef10755700e54073dd1249b769e63a6b51e8c09141 not found: ID does not exist" Jul 10 18:09:47.594512 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.594465 2582 scope.go:117] "RemoveContainer" containerID="c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2" Jul 10 18:09:47.594757 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:09:47.594739 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2\": container with ID starting with c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2 not found: ID does not exist" containerID="c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2" Jul 10 18:09:47.594800 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.594765 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2"} err="failed to get container status \"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2\": rpc error: code = NotFound desc = could not find container \"c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2\": container with ID starting with c2a4b47156163eb5f846c745129f806a235df38075c69248e9d2c564a09681d2 not found: ID does not exist" Jul 10 18:09:47.616305 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.616272 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:47.628064 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:47.628034 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f6b351bc-openshift-default-7fd59b87f9-wdrg2"] Jul 10 18:09:49.432714 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:49.432682 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" path="/var/lib/kubelet/pods/4858923c-0ae2-4b93-b0f1-782963e800c7/volumes" Jul 10 18:09:53.558355 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:53.558321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:09:53.730319 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:09:53.730288 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:00.524965 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.524933 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:00.931856 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.931825 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:10:00.932290 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.932272 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:10:00.932392 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.932318 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:10:00.932466 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.932453 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:10:00.932521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.932473 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:10:00.935696 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.935677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:00.940574 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.940556 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ba6e3fc9-openshift-default-dockercfg-z7pls\"" Jul 10 18:10:00.954676 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.954650 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:10:00.987571 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:00.987541 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:01.050741 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050702 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.050741 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.050943 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.050943 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.050943 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.050943 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.051105 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.050982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnjm\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.051105 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.051041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.051105 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.051077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.151720 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.151902 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.151902 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnjm\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.151902 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151817 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.151902 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.151903 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152111 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152135 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152452 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152355 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152519 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.152672 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.152650 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.154385 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.154365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.154598 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.154580 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.169846 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.169818 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnjm\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.191791 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.191726 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token\") pod \"e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.245602 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.245562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:01.353968 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.353942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:01.397443 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.397400 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:10:01.399827 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:10:01.399802 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ae6baf_7a2d_4304_9c8f_c51ee40a9311.slice/crio-db1ef40daee979a969bceb779936aa41438cfad0c74e7f1fddca857936c8d3a6 WatchSource:0}: Error finding container db1ef40daee979a969bceb779936aa41438cfad0c74e7f1fddca857936c8d3a6: Status 404 returned error can't find the container with id db1ef40daee979a969bceb779936aa41438cfad0c74e7f1fddca857936c8d3a6 Jul 10 18:10:01.401935 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.401895 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:01.402030 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.401983 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:01.402077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.402056 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:01.642314 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.642268 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerStarted","Data":"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562"} Jul 10 18:10:01.642708 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.642338 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerStarted","Data":"db1ef40daee979a969bceb779936aa41438cfad0c74e7f1fddca857936c8d3a6"} Jul 10 18:10:01.669756 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:01.669683 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podStartSLOduration=1.669668276 podStartE2EDuration="1.669668276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:10:01.667984424 +0000 UTC m=+1868.728799787" watchObservedRunningTime="2026-07-10 18:10:01.669668276 +0000 UTC m=+1868.730483881" Jul 10 18:10:02.246190 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.246163 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:02.251125 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.251100 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:02.647291 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.647198 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:02.648099 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.648077 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:02.727229 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.727193 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:02.828559 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:02.828531 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:03.774818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:03.774769 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:04.075574 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:04.075493 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:04.184762 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:04.184728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:04.298427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:04.298397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:04.737521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:04.737488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:04.835746 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:04.835717 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:08.179231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:08.179180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:12.779019 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:12.778963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:17.649451 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.649357 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 18:10:17.649451 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.649440 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 18:10:17.735150 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.735122 2582 generic.go:356] "Generic (PLEG): container finished" podID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerID="3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562" exitCode=0 Jul 10 18:10:17.735324 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.735202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerDied","Data":"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562"} Jul 10 18:10:17.735598 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.735585 2582 scope.go:117] "RemoveContainer" containerID="3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562" Jul 10 18:10:17.736587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.736557 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:17.736638 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.736626 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:17.736684 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:17.736660 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:10:18.747729 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:18.747689 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerStarted","Data":"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493"} Jul 10 18:10:19.246246 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:19.246211 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:19.246506 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:19.246483 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 18:10:19.246554 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:19.246536 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 18:10:20.246455 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:20.246415 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 18:10:20.246848 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:20.246478 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 18:10:21.246490 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:21.246453 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 18:10:21.246889 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:21.246522 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 18:10:22.246386 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:22.246351 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 10 18:10:22.246572 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:22.246410 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 10 18:10:23.250175 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:23.250148 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:23.250574 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:23.250471 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:23.251215 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:23.251196 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:10:40.848176 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:40.848128 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:42.304061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:42.304019 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:46.860344 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:46.860309 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:10:51.930924 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:10:51.930886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:28.526404 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:28.526366 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:29.476440 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:29.476411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:35.101042 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.100980 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:35.204208 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.204175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:35.468925 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.468896 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:11:35.469133 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.469104 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" containerID="cri-o://fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493" gracePeriod=30 Jul 10 18:11:35.504645 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.504611 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:35.703603 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:35.703566 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:36.776551 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:36.776523 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:36.908626 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:36.908593 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:37.032055 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:37.031953 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:37.827615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:37.827585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:37.925842 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:37.925809 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:38.025976 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:38.025942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:38.252667 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:38.252630 2582 patch_prober.go:28] interesting pod/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 18:11:38.252840 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:38.252711 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 18:11:39.534528 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:39.534494 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:40.729753 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.729730 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:11:40.882825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882730 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.882825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882770 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.882825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhnjm\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883154 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882839 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883154 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882917 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883154 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.882957 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883154 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883078 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883154 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.883426 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883193 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket\") pod \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\" (UID: \"98ae6baf-7a2d-4304-9c8f-c51ee40a9311\") " Jul 10 18:11:40.884137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:11:40.884137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883823 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:40.884137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.883884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data" (OuterVolumeSpecName: "istio-data") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:40.884137 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.884089 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:40.884439 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.884324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:40.885665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.885638 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:11:40.885794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.885661 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm" (OuterVolumeSpecName: "kube-api-access-bhnjm") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "kube-api-access-bhnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:40.885865 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.885827 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:11:40.886202 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.886183 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token" (OuterVolumeSpecName: "istio-token") pod "98ae6baf-7a2d-4304-9c8f-c51ee40a9311" (UID: "98ae6baf-7a2d-4304-9c8f-c51ee40a9311"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:11:40.984396 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984364 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984396 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984392 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984396 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984401 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984410 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnjm\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-kube-api-access-bhnjm\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984418 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984427 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984434 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984441 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:40.984618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:40.984449 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ae6baf-7a2d-4304-9c8f-c51ee40a9311-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:11:41.132980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.132894 2582 generic.go:356] "Generic (PLEG): container finished" podID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerID="fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493" exitCode=0 Jul 10 18:11:41.132980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.132961 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" Jul 10 18:11:41.132980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.132974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerDied","Data":"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493"} Jul 10 18:11:41.133231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.133071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg" event={"ID":"98ae6baf-7a2d-4304-9c8f-c51ee40a9311","Type":"ContainerDied","Data":"db1ef40daee979a969bceb779936aa41438cfad0c74e7f1fddca857936c8d3a6"} Jul 10 18:11:41.133231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.133094 2582 scope.go:117] "RemoveContainer" containerID="fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493" Jul 10 18:11:41.142366 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.142352 2582 scope.go:117] "RemoveContainer" containerID="3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562" Jul 10 18:11:41.150416 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.150399 2582 scope.go:117] "RemoveContainer" containerID="fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493" Jul 10 18:11:41.150654 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:11:41.150639 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493\": container with ID starting with fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493 not found: ID does not exist" containerID="fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493" Jul 10 18:11:41.150694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.150662 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493"} err="failed to get container status \"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493\": rpc error: code = NotFound desc = could not find container \"fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493\": container with ID starting with fea6edc27f16c29d86bdccf9437255aea5179bd2604e6686a530a3a85b954493 not found: ID does not exist" Jul 10 18:11:41.150694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.150678 2582 scope.go:117] "RemoveContainer" containerID="3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562" Jul 10 18:11:41.150900 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:11:41.150884 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562\": container with ID starting with 3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562 not found: ID does not exist" containerID="3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562" Jul 10 18:11:41.150939 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.150907 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562"} err="failed to get container status \"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562\": rpc error: code = NotFound desc = could not find container \"3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562\": container with ID starting with 3c3f59dfde02dbe8cebd7fb039d600652b4936b5fbde192e42e8397dff10d562 not found: ID does not exist" Jul 10 18:11:41.155639 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.155618 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:11:41.165654 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.165630 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ba6e3fc9-openshift-default-746fb747d7-qzzgg"] Jul 10 18:11:41.433300 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:41.433267 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" path="/var/lib/kubelet/pods/98ae6baf-7a2d-4304-9c8f-c51ee40a9311/volumes" Jul 10 18:11:45.331257 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:45.331184 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:45.431388 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:45.431361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:46.078615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:46.078585 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:52.029216 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:52.029176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:52.102683 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:52.102645 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:52.198643 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:52.198610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:11:52.301710 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:11:52.301627 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:02.388214 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:02.388176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:02.489139 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:02.489104 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:02.851056 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:02.851009 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:03.178268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:03.178236 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:24.183839 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:24.183800 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:30.402322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:30.402282 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:30.502167 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:30.502139 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:30.843109 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:30.843016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:31.119061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.118961 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:12:31.119383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119369 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:12:31.119453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119385 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4858923c-0ae2-4b93-b0f1-782963e800c7" containerName="istio-proxy" Jul 10 18:12:31.119453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119399 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.119453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119405 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.119453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119413 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.119453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119419 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.119633 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119500 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.119633 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.119513 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ae6baf-7a2d-4304-9c8f-c51ee40a9311" containerName="istio-proxy" Jul 10 18:12:31.122558 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.122542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.125157 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.125135 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b1e501-openshift-default-dockercfg-2btlj\"" Jul 10 18:12:31.136484 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.136440 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:31.137483 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.137462 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:12:31.227421 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227382 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227490 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227875 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227640 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227875 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227875 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.227875 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.227757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2rz\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328590 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328556 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328590 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2rz\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.328799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.328839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329205 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.329084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329263 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.329215 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329309 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.329266 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.329365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.329572 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.329556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.331157 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.331136 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.331268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.331233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.338013 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.337968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.338394 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.338373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2rz\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz\") pod \"e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.404423 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.404397 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:31.432953 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.432928 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:31.571786 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.571763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:12:31.573376 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:12:31.573346 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode337fce6_82c9_407f_bb2a_5f249d3a2153.slice/crio-d0211b3d42b20ae2329fcb6cc456acb66468bf4b47ba67fed60cae698f83c14b WatchSource:0}: Error finding container d0211b3d42b20ae2329fcb6cc456acb66468bf4b47ba67fed60cae698f83c14b: Status 404 returned error can't find the container with id d0211b3d42b20ae2329fcb6cc456acb66468bf4b47ba67fed60cae698f83c14b Jul 10 18:12:31.575598 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.575560 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:31.575678 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.575647 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:31.575717 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:31.575691 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:32.371603 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:32.371572 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerStarted","Data":"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145"} Jul 10 18:12:32.371790 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:32.371626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerStarted","Data":"d0211b3d42b20ae2329fcb6cc456acb66468bf4b47ba67fed60cae698f83c14b"} Jul 10 18:12:32.398457 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:32.398401 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podStartSLOduration=1.398380543 podStartE2EDuration="1.398380543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:12:32.397272134 +0000 UTC m=+2019.458087470" watchObservedRunningTime="2026-07-10 18:12:32.398380543 +0000 UTC m=+2019.459195881" Jul 10 18:12:32.433709 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:32.433672 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:32.438544 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:32.438521 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:33.376642 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:33.376607 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:33.377547 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:33.377527 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:34.469118 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:34.469074 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:34.594339 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:34.594302 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:34.691450 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:34.691413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:34.829513 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:34.829429 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:35.302427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:35.302391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:35.402728 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:35.402695 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:38.950167 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:38.950135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:39.306097 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.305978 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:39.642041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.641926 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:12:39.645629 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.645612 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.648908 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.648885 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3fac88-openshift-default-dockercfg-74j8p\"" Jul 10 18:12:39.673329 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.673306 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:12:39.706204 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706204 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706552 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706437 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.706552 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.706454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5pr\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.729490 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.729455 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:39.806950 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.806915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.806950 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.806952 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.806971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807041 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5pr\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807232 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807589 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807406 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807589 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807436 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807589 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807556 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.807912 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.808022 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.807973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.809400 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.809381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.809732 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.809701 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.815818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.815795 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5pr\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.816160 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.816142 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token\") pod \"e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:39.918574 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.918548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:39.956687 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:39.956662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:40.099465 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.099383 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:12:40.101956 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:12:40.101928 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6255952_a81f_4a16_9e45_8bdd76310f4b.slice/crio-04a8991b240cff760a0e8d523ae6a609b456b6d3d32001826185ea56370e71dc WatchSource:0}: Error finding container 04a8991b240cff760a0e8d523ae6a609b456b6d3d32001826185ea56370e71dc: Status 404 returned error can't find the container with id 04a8991b240cff760a0e8d523ae6a609b456b6d3d32001826185ea56370e71dc Jul 10 18:12:40.104431 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.104398 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:40.104521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.104464 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:40.104521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.104491 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:40.412314 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.412277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerStarted","Data":"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a"} Jul 10 18:12:40.412484 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.412335 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerStarted","Data":"04a8991b240cff760a0e8d523ae6a609b456b6d3d32001826185ea56370e71dc"} Jul 10 18:12:40.440908 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.440860 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podStartSLOduration=1.440840581 podStartE2EDuration="1.440840581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:12:40.439834541 +0000 UTC m=+2027.500649876" watchObservedRunningTime="2026-07-10 18:12:40.440840581 +0000 UTC m=+2027.501655921" Jul 10 18:12:40.957848 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:40.957813 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:41.962454 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:41.962428 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:42.422318 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.422288 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:42.423389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.423366 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:12:42.439405 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.439383 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:42.553412 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.553377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:42.650818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.650789 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:42.764126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:42.764016 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:43.222341 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:43.222280 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:43.319073 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:43.318986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:46.936444 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:46.936409 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:49.419179 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:49.419143 2582 patch_prober.go:28] interesting pod/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 18:12:49.419574 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:49.419208 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 18:12:50.466891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.466855 2582 generic.go:356] "Generic (PLEG): container finished" podID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerID="2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145" exitCode=0 Jul 10 18:12:50.467387 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.466925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerDied","Data":"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145"} Jul 10 18:12:50.467387 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.467336 2582 scope.go:117] "RemoveContainer" containerID="2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145" Jul 10 18:12:50.468294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.468265 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:50.468374 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.468337 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:50.468374 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:50.468368 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:51.477508 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:51.477460 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerStarted","Data":"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c"} Jul 10 18:12:52.434354 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:52.434317 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:52.434711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:52.434683 2582 patch_prober.go:28] interesting pod/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:12:52.434888 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:52.434859 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 18:12:53.434601 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:53.434561 2582 patch_prober.go:28] interesting pod/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:12:53.435097 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:53.434627 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 18:12:54.433680 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:54.433620 2582 patch_prober.go:28] interesting pod/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:12:54.433976 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:54.433945 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 18:12:55.437284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:55.437249 2582 patch_prober.go:28] interesting pod/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 10 18:12:55.437870 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:55.437841 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 10 18:12:56.247341 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:56.247305 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:56.344731 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:56.344691 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:12:56.436912 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:56.436883 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:56.437292 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:56.437235 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:56.438084 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:56.438066 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:12:58.464182 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:58.464146 2582 patch_prober.go:28] interesting pod/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 18:12:58.465395 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:58.464216 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 18:12:59.526968 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.526933 2582 generic.go:356] "Generic (PLEG): container finished" podID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerID="a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a" exitCode=0 Jul 10 18:12:59.527376 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.527022 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerDied","Data":"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a"} Jul 10 18:12:59.527422 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.527401 2582 scope.go:117] "RemoveContainer" containerID="a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a" Jul 10 18:12:59.528431 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.528399 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:59.528542 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.528506 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:12:59.528605 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:12:59.528558 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:13:00.533639 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:00.533603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerStarted","Data":"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727"} Jul 10 18:13:00.957890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:00.957854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:13:00.958088 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:00.958061 2582 patch_prober.go:28] interesting pod/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 18:13:00.958135 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:00.958110 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 18:13:01.958067 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:01.958031 2582 patch_prober.go:28] interesting pod/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 18:13:01.958473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:01.958084 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 18:13:02.957875 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:02.957837 2582 patch_prober.go:28] interesting pod/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 18:13:02.958082 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:02.957892 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 18:13:03.957675 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:03.957640 2582 patch_prober.go:28] interesting pod/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 10 18:13:03.958092 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:03.957697 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 10 18:13:04.032317 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:04.032268 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:13:04.133100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:04.133054 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:13:04.961184 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:04.961155 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:13:04.961723 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:04.961427 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:13:04.962228 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:04.962210 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:13:08.856591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:08.856554 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:13:35.603612 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:35.603582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:13:53.484442 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:53.484416 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:13:53.490234 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:53.490214 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:13:55.687922 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:13:55.687884 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:01.177920 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:01.177886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:02.114071 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:02.114042 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:02.267294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:02.267255 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:02.684544 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:02.684508 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:02.848897 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:02.848862 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:02.946246 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:02.946160 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:03.113541 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:03.113510 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:07.754056 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:07.754018 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:45.180882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.180745 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:45.425464 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.425421 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:14:45.425771 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.425743 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" containerID="cri-o://b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c" gracePeriod=30 Jul 10 18:14:45.440248 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.440176 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:45.690556 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.690469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:45.994262 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:45.994167 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:46.322224 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:46.322146 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:46.521306 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:46.521276 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:46.622460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:46.622377 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:46.814537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:46.814498 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:50.684330 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.684306 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:14:50.761874 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.761874 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761824 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761882 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761904 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761931 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.761963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762369 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762337 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:50.762421 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762322 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:50.762421 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762375 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:50.762498 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762415 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:14:50.762498 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762430 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762498 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762489 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762599 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762579 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd2rz\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz\") pod \"e337fce6-82c9-407f-bb2a-5f249d3a2153\" (UID: \"e337fce6-82c9-407f-bb2a-5f249d3a2153\") " Jul 10 18:14:50.762881 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762860 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.762881 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762883 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.763077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762896 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e337fce6-82c9-407f-bb2a-5f249d3a2153-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.763077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.762907 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.763148 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.763077 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data" (OuterVolumeSpecName: "istio-data") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:50.764354 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.764330 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:50.764530 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.764505 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token" (OuterVolumeSpecName: "istio-token") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:50.764659 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.764641 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:14:50.764885 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.764868 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz" (OuterVolumeSpecName: "kube-api-access-kd2rz") pod "e337fce6-82c9-407f-bb2a-5f249d3a2153" (UID: "e337fce6-82c9-407f-bb2a-5f249d3a2153"). InnerVolumeSpecName "kube-api-access-kd2rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:50.863381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.863344 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.863381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.863371 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.863381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.863379 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2rz\" (UniqueName: \"kubernetes.io/projected/e337fce6-82c9-407f-bb2a-5f249d3a2153-kube-api-access-kd2rz\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.863381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.863388 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:50.863659 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:50.863396 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e337fce6-82c9-407f-bb2a-5f249d3a2153-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:51.071125 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.071031 2582 generic.go:356] "Generic (PLEG): container finished" podID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerID="b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c" exitCode=0 Jul 10 18:14:51.071125 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.071091 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" Jul 10 18:14:51.071125 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.071110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerDied","Data":"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c"} Jul 10 18:14:51.071404 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.071185 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx" event={"ID":"e337fce6-82c9-407f-bb2a-5f249d3a2153","Type":"ContainerDied","Data":"d0211b3d42b20ae2329fcb6cc456acb66468bf4b47ba67fed60cae698f83c14b"} Jul 10 18:14:51.071404 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.071206 2582 scope.go:117] "RemoveContainer" containerID="b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c" Jul 10 18:14:51.080646 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.080626 2582 scope.go:117] "RemoveContainer" containerID="2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145" Jul 10 18:14:51.088525 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.088508 2582 scope.go:117] "RemoveContainer" containerID="b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c" Jul 10 18:14:51.088782 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:14:51.088764 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c\": container with ID starting with b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c not found: ID does not exist" containerID="b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c" Jul 10 18:14:51.088831 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.088791 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c"} err="failed to get container status \"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c\": rpc error: code = NotFound desc = could not find container \"b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c\": container with ID starting with b57cbf71d369f74fcb58effbf839533fc9e8d4579bccfacf2694e7286fdfcc7c not found: ID does not exist" Jul 10 18:14:51.088831 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.088808 2582 scope.go:117] "RemoveContainer" containerID="2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145" Jul 10 18:14:51.089048 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:14:51.089032 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145\": container with ID starting with 2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145 not found: ID does not exist" containerID="2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145" Jul 10 18:14:51.089093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.089054 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145"} err="failed to get container status \"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145\": rpc error: code = NotFound desc = could not find container \"2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145\": container with ID starting with 2f9e4ca1a68ee9ad79cd689da59f9053ccc1964a079e4d7880e4eda9643cd145 not found: ID does not exist" Jul 10 18:14:51.100281 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.100250 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:14:51.100413 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.100302 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1e501-openshift-default-6f58cb7465-h8swx"] Jul 10 18:14:51.433524 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:51.433491 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" path="/var/lib/kubelet/pods/e337fce6-82c9-407f-bb2a-5f249d3a2153/volumes" Jul 10 18:14:52.334805 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.334760 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:52.435383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.435322 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:52.604685 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.604598 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:14:52.604886 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.604825 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" containerID="cri-o://6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727" gracePeriod=30 Jul 10 18:14:52.631197 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.631161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:52.863530 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:52.863439 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:53.237315 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.237277 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:53.560555 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560484 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:14:53.560896 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560840 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:14:53.560896 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560850 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:14:53.560896 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560864 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:14:53.560896 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560871 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:14:53.561062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.560929 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:14:53.565408 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.565387 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.568234 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.568203 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-dfa29f-openshift-default-dockercfg-fp2fr\"" Jul 10 18:14:53.578949 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.578924 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:14:53.581885 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.581863 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:53.586438 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586570 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586570 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586490 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586570 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586743 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586577 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586743 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586620 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586743 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586743 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586698 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7r6c\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.586906 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.586763 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687745 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687824 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687875 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.687913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.687910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688175 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688175 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688175 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7r6c\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688821 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.688821 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.689019 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.688978 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.690397 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.690370 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.690531 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.690518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.696929 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.696910 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.697256 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.697195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7r6c\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c\") pod \"e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:53.804380 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.804346 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:53.876141 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:53.876059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:54.024498 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.024472 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:14:54.025531 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:14:54.025495 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbad365_a07a_4f99_9679_f0acf0ac0a5f.slice/crio-ce1dac0be23f61dd8c43f4786f6e82fb9ef58ecdab8ec53c11d261b6338b1a23 WatchSource:0}: Error finding container ce1dac0be23f61dd8c43f4786f6e82fb9ef58ecdab8ec53c11d261b6338b1a23: Status 404 returned error can't find the container with id ce1dac0be23f61dd8c43f4786f6e82fb9ef58ecdab8ec53c11d261b6338b1a23 Jul 10 18:14:54.028195 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.028164 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:14:54.028269 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.028221 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:14:54.028269 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.028250 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:14:54.095231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.095201 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerStarted","Data":"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015"} Jul 10 18:14:54.095410 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.095291 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerStarted","Data":"ce1dac0be23f61dd8c43f4786f6e82fb9ef58ecdab8ec53c11d261b6338b1a23"} Jul 10 18:14:54.117977 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.117914 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podStartSLOduration=1.117892866 podStartE2EDuration="1.117892866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:14:54.115405923 +0000 UTC m=+2161.176221259" watchObservedRunningTime="2026-07-10 18:14:54.117892866 +0000 UTC m=+2161.178708202" Jul 10 18:14:54.877222 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.877186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:54.932976 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:54.932944 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:55.881010 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:55.880964 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:56.105011 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.104965 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:56.105831 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.105809 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:14:56.218165 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.218135 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:56.349970 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.349936 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:56.568440 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.568338 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:56.766634 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:56.766597 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:57.080521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.080489 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:57.181225 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.181196 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:14:57.857340 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.857314 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:14:57.925716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925690 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925792 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925809 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925831 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925851 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r5pr\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.925891 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.925882 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.926316 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.926067 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo\") pod \"a6255952-a81f-4a16-9e45-8bdd76310f4b\" (UID: \"a6255952-a81f-4a16-9e45-8bdd76310f4b\") " Jul 10 18:14:57.926316 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.926130 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:14:57.926442 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.926420 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6255952-a81f-4a16-9e45-8bdd76310f4b-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:57.926648 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.926530 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:57.926739 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.926673 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:57.927077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.927044 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:57.927350 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.927108 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:57.928643 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.928621 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr" (OuterVolumeSpecName: "kube-api-access-5r5pr") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "kube-api-access-5r5pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:57.928898 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.928876 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:14:57.928979 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.928877 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:14:57.929121 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:57.929098 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6255952-a81f-4a16-9e45-8bdd76310f4b" (UID: "a6255952-a81f-4a16-9e45-8bdd76310f4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:14:58.027061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027028 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027055 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027061 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027065 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027074 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027083 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027091 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r5pr\" (UniqueName: \"kubernetes.io/projected/a6255952-a81f-4a16-9e45-8bdd76310f4b-kube-api-access-5r5pr\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027099 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6255952-a81f-4a16-9e45-8bdd76310f4b-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.027294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.027107 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6255952-a81f-4a16-9e45-8bdd76310f4b-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:14:58.115581 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.115545 2582 generic.go:356] "Generic (PLEG): container finished" podID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerID="6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727" exitCode=0 Jul 10 18:14:58.116100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.115608 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" Jul 10 18:14:58.116100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.115632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerDied","Data":"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727"} Jul 10 18:14:58.116100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.115688 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg" event={"ID":"a6255952-a81f-4a16-9e45-8bdd76310f4b","Type":"ContainerDied","Data":"04a8991b240cff760a0e8d523ae6a609b456b6d3d32001826185ea56370e71dc"} Jul 10 18:14:58.116100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.115715 2582 scope.go:117] "RemoveContainer" containerID="6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727" Jul 10 18:14:58.125159 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.125142 2582 scope.go:117] "RemoveContainer" containerID="a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a" Jul 10 18:14:58.132900 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.132883 2582 scope.go:117] "RemoveContainer" containerID="6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727" Jul 10 18:14:58.133198 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:14:58.133179 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727\": container with ID starting with 6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727 not found: ID does not exist" containerID="6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727" Jul 10 18:14:58.133248 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.133205 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727"} err="failed to get container status \"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727\": rpc error: code = NotFound desc = could not find container \"6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727\": container with ID starting with 6370c3b2d9a00c192bc0b7fa097005ca57e4e7bf43ef841fe3c038fb7fd34727 not found: ID does not exist" Jul 10 18:14:58.133248 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.133224 2582 scope.go:117] "RemoveContainer" containerID="a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a" Jul 10 18:14:58.133471 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:14:58.133451 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a\": container with ID starting with a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a not found: ID does not exist" containerID="a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a" Jul 10 18:14:58.133517 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.133477 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a"} err="failed to get container status \"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a\": rpc error: code = NotFound desc = could not find container \"a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a\": container with ID starting with a4c5830171d21419cb84bc014978e2cb9ac78a56275f662f2a6f603204dff57a not found: ID does not exist" Jul 10 18:14:58.136962 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.136937 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:14:58.141093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:58.141072 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3fac88-openshift-default-d5d4fdd45-w44zg"] Jul 10 18:14:59.433306 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:14:59.433272 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" path="/var/lib/kubelet/pods/a6255952-a81f-4a16-9e45-8bdd76310f4b/volumes" Jul 10 18:15:00.139727 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.139691 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:00.140093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140079 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.140093 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140094 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.140230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140102 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.140230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140108 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.140230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140182 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.140230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.140192 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e337fce6-82c9-407f-bb2a-5f249d3a2153" containerName="istio-proxy" Jul 10 18:15:00.144918 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.144895 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:15:00.145074 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.145054 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" Jul 10 18:15:00.145426 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.145410 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6255952-a81f-4a16-9e45-8bdd76310f4b" containerName="istio-proxy" Jul 10 18:15:00.147607 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.147589 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jxczz\"" Jul 10 18:15:00.148321 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.148305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" Jul 10 18:15:00.151401 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.151347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:00.155084 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.155054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:15:00.158273 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.158252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" Jul 10 18:15:00.165767 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.165751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" Jul 10 18:15:00.314359 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.314332 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:00.315362 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:15:00.315333 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6db12b_14c6_4e5c_bdcf_9e4cbd6ca61b.slice/crio-376b85c5f60ce61f324f3debde4718d6070909e3b1cb2227ca1d14369371545c WatchSource:0}: Error finding container 376b85c5f60ce61f324f3debde4718d6070909e3b1cb2227ca1d14369371545c: Status 404 returned error can't find the container with id 376b85c5f60ce61f324f3debde4718d6070909e3b1cb2227ca1d14369371545c Jul 10 18:15:00.340133 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.339915 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:15:00.342101 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:15:00.342076 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a27f047_db08_421b_adfe_bb961e188b81.slice/crio-fb9cc05b0150504c698a49df681d74dd7ed10375417305774f37ade3c1f87b74 WatchSource:0}: Error finding container fb9cc05b0150504c698a49df681d74dd7ed10375417305774f37ade3c1f87b74: Status 404 returned error can't find the container with id fb9cc05b0150504c698a49df681d74dd7ed10375417305774f37ade3c1f87b74 Jul 10 18:15:00.616134 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:00.616097 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:01.113813 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.113778 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:01.132284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.132246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" event={"ID":"4a27f047-db08-421b-adfe-bb961e188b81","Type":"ContainerStarted","Data":"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5"} Jul 10 18:15:01.132392 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.132306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" event={"ID":"4a27f047-db08-421b-adfe-bb961e188b81","Type":"ContainerStarted","Data":"fb9cc05b0150504c698a49df681d74dd7ed10375417305774f37ade3c1f87b74"} Jul 10 18:15:01.133842 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.133821 2582 generic.go:356] "Generic (PLEG): container finished" podID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerID="9933989934c12b8dd29455688e5f48381216afa942464fe3fcb67fc524f7590d" exitCode=7 Jul 10 18:15:01.133915 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.133861 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerDied","Data":"9933989934c12b8dd29455688e5f48381216afa942464fe3fcb67fc524f7590d"} Jul 10 18:15:01.133915 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.133891 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerStarted","Data":"376b85c5f60ce61f324f3debde4718d6070909e3b1cb2227ca1d14369371545c"} Jul 10 18:15:01.134029 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.134011 2582 scope.go:117] "RemoveContainer" containerID="9933989934c12b8dd29455688e5f48381216afa942464fe3fcb67fc524f7590d" Jul 10 18:15:01.153078 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.153039 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" podStartSLOduration=1.153025151 podStartE2EDuration="1.153025151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:15:01.15115467 +0000 UTC m=+2168.211970006" watchObservedRunningTime="2026-07-10 18:15:01.153025151 +0000 UTC m=+2168.213840486" Jul 10 18:15:01.420540 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.420508 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:01.423887 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.423871 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.426152 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.426130 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ea3624-openshift-default-dockercfg-lqp7v\"" Jul 10 18:15:01.440785 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.440761 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:01.453708 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.453683 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:01.458818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.458799 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.458898 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.458830 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.458898 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.458853 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.458898 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.458870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.459041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.458917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.459041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.459021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrfn\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.459128 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.459059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.459128 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.459116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.459207 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.459151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559776 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559967 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559967 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559967 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559967 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559926 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.559967 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560261 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.559984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560261 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560261 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrfn\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560261 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560233 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560443 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560332 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560502 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.560615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.560615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.562540 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.562520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.562733 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.562714 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.571971 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.571948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.572081 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.572004 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrfn\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn\") pod \"e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.735267 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.735183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:01.767122 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.767090 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:01.884446 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.884422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:01.886659 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:15:01.886632 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4724bc0_44d8_4f9f_aafe_c9268631659c.slice/crio-23c9d8c28b9f7df73e78000a83d2a8cf9a9a6dfdc6afa43c8ea9bb6f8b88f94f WatchSource:0}: Error finding container 23c9d8c28b9f7df73e78000a83d2a8cf9a9a6dfdc6afa43c8ea9bb6f8b88f94f: Status 404 returned error can't find the container with id 23c9d8c28b9f7df73e78000a83d2a8cf9a9a6dfdc6afa43c8ea9bb6f8b88f94f Jul 10 18:15:01.888703 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.888669 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:01.888777 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.888739 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:01.888777 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:01.888763 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:02.147687 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.147590 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerStarted","Data":"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5"} Jul 10 18:15:02.147873 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.147692 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerStarted","Data":"23c9d8c28b9f7df73e78000a83d2a8cf9a9a6dfdc6afa43c8ea9bb6f8b88f94f"} Jul 10 18:15:02.149708 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.149681 2582 generic.go:356] "Generic (PLEG): container finished" podID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerID="11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716" exitCode=7 Jul 10 18:15:02.149836 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.149736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerDied","Data":"11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716"} Jul 10 18:15:02.149836 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.149780 2582 scope.go:117] "RemoveContainer" containerID="9933989934c12b8dd29455688e5f48381216afa942464fe3fcb67fc524f7590d" Jul 10 18:15:02.149950 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.149905 2582 scope.go:117] "RemoveContainer" containerID="11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716" Jul 10 18:15:02.150190 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:02.150165 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f_odh-ai-gateway-infra(db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" Jul 10 18:15:02.173919 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.173881 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podStartSLOduration=1.173866984 podStartE2EDuration="1.173866984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:15:02.171772838 +0000 UTC m=+2169.232588175" watchObservedRunningTime="2026-07-10 18:15:02.173866984 +0000 UTC m=+2169.234682318" Jul 10 18:15:02.735827 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.735788 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:02.740619 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:02.740582 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:03.156364 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:03.156339 2582 scope.go:117] "RemoveContainer" containerID="11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716" Jul 10 18:15:03.156549 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:03.156509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:03.156611 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:03.156580 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f_odh-ai-gateway-infra(db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" Jul 10 18:15:03.157540 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:03.157522 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:03.604951 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:03.604871 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:03.705365 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:03.705329 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:04.241665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:04.241638 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:04.338421 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:04.338391 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:04.428463 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:04.428418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:04.684742 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:04.684713 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:05.118553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:05.118469 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:05.289077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:05.289047 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:08.679095 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:08.679064 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:12.146189 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:12.146147 2582 patch_prober.go:28] interesting pod/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 18:15:12.146561 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:12.146213 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 18:15:12.316917 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:12.316881 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:14.215585 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.215553 2582 generic.go:356] "Generic (PLEG): container finished" podID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerID="35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015" exitCode=0 Jul 10 18:15:14.216056 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.215628 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerDied","Data":"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015"} Jul 10 18:15:14.216128 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.216062 2582 scope.go:117] "RemoveContainer" containerID="35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015" Jul 10 18:15:14.217079 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.217054 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:14.217141 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.217119 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:14.217192 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.217150 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:14.427219 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:14.427183 2582 scope.go:117] "RemoveContainer" containerID="11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716" Jul 10 18:15:15.226408 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.225410 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerStarted","Data":"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68"} Jul 10 18:15:15.229345 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.229318 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerStarted","Data":"863d9fb8e27fef245ed7214e16004e52d995ca17a5c38a07343fb6d0573bbf55"} Jul 10 18:15:15.265717 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.265670 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" podStartSLOduration=15.265654059 podStartE2EDuration="15.265654059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:15:15.265585692 +0000 UTC m=+2182.326401027" watchObservedRunningTime="2026-07-10 18:15:15.265654059 +0000 UTC m=+2182.326469395" Jul 10 18:15:15.877306 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.877272 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:15.877488 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.877366 2582 patch_prober.go:28] interesting pod/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 18:15:15.877488 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:15.877402 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 18:15:16.276263 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:16.276227 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:16.276761 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:16.276407 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" containerID="cri-o://863d9fb8e27fef245ed7214e16004e52d995ca17a5c38a07343fb6d0573bbf55" gracePeriod=30 Jul 10 18:15:16.874786 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:16.874750 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:16.876649 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:16.876629 2582 patch_prober.go:28] interesting pod/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 18:15:16.876706 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:16.876676 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 18:15:17.295890 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.295855 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:17.395953 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.395921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:17.495694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.495661 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:17.640751 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.640643 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:17.876459 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.876421 2582 patch_prober.go:28] interesting pod/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 18:15:17.876646 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:17.876477 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 18:15:18.876830 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:18.876794 2582 patch_prober.go:28] interesting pod/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 10 18:15:18.877237 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:18.876848 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 10 18:15:19.882265 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:19.882236 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:19.882736 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:19.882619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:19.883360 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:19.883340 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:22.271587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.271556 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerID="b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5" exitCode=0 Jul 10 18:15:22.271970 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.271631 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerDied","Data":"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5"} Jul 10 18:15:22.272060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.272044 2582 scope.go:117] "RemoveContainer" containerID="b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5" Jul 10 18:15:22.273149 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.273119 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:22.273226 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.273208 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:22.273274 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:22.273251 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:23.278087 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.278053 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerStarted","Data":"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f"} Jul 10 18:15:23.430166 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.430133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:23.504586 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.504545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:23.678840 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.678810 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:15:23.679092 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.679064 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" containerID="cri-o://75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68" gracePeriod=30 Jul 10 18:15:23.714161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.714105 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:23.735431 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.735397 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:23.735745 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.735711 2582 patch_prober.go:28] interesting pod/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 18:15:23.735868 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.735778 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 18:15:23.931874 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:23.931731 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:24.123928 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.123893 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:24.523381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.523252 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:24.628216 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.628185 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:24.726451 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.726396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:24.735665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.735637 2582 patch_prober.go:28] interesting pod/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 18:15:24.735860 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.735683 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 18:15:24.892656 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:24.892583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:25.736630 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:25.736591 2582 patch_prober.go:28] interesting pod/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 18:15:25.737038 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:25.736664 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 18:15:26.735697 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:26.735653 2582 patch_prober.go:28] interesting pod/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 18:15:26.735886 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:26.735735 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 18:15:27.739529 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:27.739499 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:27.740041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:27.739832 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:27.740460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:27.740443 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:28.932651 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.932628 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:28.954731 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954643 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.954731 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954688 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.954731 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.954982 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954877 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.954982 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954930 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.955126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.954980 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:28.955126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955014 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7r6c\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.955126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955080 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.955126 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955114 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.955327 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955142 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert\") pod \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\" (UID: \"0cbad365-a07a-4f99-9679-f0acf0ac0a5f\") " Jul 10 18:15:28.955526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955488 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:28.955920 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955541 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:28.955920 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955669 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:28.955920 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.955836 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:15:28.956368 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.956334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:28.957643 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.957623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:28.957738 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.957638 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c" (OuterVolumeSpecName: "kube-api-access-s7r6c") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "kube-api-access-s7r6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:15:28.957871 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.957848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:15:28.958359 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:28.958334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cbad365-a07a-4f99-9679-f0acf0ac0a5f" (UID: "0cbad365-a07a-4f99-9679-f0acf0ac0a5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:15:29.056460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056414 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056454 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056460 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056466 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056705 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056478 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056705 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056493 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7r6c\" (UniqueName: \"kubernetes.io/projected/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-kube-api-access-s7r6c\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056705 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056504 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056705 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056515 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.056705 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.056525 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cbad365-a07a-4f99-9679-f0acf0ac0a5f-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:29.308956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.308853 2582 generic.go:356] "Generic (PLEG): container finished" podID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerID="75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68" exitCode=0 Jul 10 18:15:29.308956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.308897 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" Jul 10 18:15:29.308956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.308929 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerDied","Data":"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68"} Jul 10 18:15:29.309248 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.309007 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs" event={"ID":"0cbad365-a07a-4f99-9679-f0acf0ac0a5f","Type":"ContainerDied","Data":"ce1dac0be23f61dd8c43f4786f6e82fb9ef58ecdab8ec53c11d261b6338b1a23"} Jul 10 18:15:29.309248 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.309025 2582 scope.go:117] "RemoveContainer" containerID="75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68" Jul 10 18:15:29.318320 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.318301 2582 scope.go:117] "RemoveContainer" containerID="35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015" Jul 10 18:15:29.326225 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.326208 2582 scope.go:117] "RemoveContainer" containerID="75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68" Jul 10 18:15:29.326469 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:29.326454 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68\": container with ID starting with 75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68 not found: ID does not exist" containerID="75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68" Jul 10 18:15:29.326534 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.326475 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68"} err="failed to get container status \"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68\": rpc error: code = NotFound desc = could not find container \"75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68\": container with ID starting with 75029b18eee6a75b48ab257fc1b95115a9c50f72b85fd69ff0cd541167387c68 not found: ID does not exist" Jul 10 18:15:29.326534 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.326490 2582 scope.go:117] "RemoveContainer" containerID="35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015" Jul 10 18:15:29.326712 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:29.326694 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015\": container with ID starting with 35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015 not found: ID does not exist" containerID="35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015" Jul 10 18:15:29.326752 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.326719 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015"} err="failed to get container status \"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015\": rpc error: code = NotFound desc = could not find container \"35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015\": container with ID starting with 35fafc20a009bf1ce450c50b92716c272a2e3c17da6d6c4899729e99524e4015 not found: ID does not exist" Jul 10 18:15:29.331656 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.331634 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:15:29.338247 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.338226 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-dfa29f-openshift-default-7bb495c6bc-crszs"] Jul 10 18:15:29.433188 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:29.433159 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" path="/var/lib/kubelet/pods/0cbad365-a07a-4f99-9679-f0acf0ac0a5f/volumes" Jul 10 18:15:30.495671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:30.495642 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:30.594381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:30.594344 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:30.772079 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:30.771981 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:30.772229 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:30.772199 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" containerID="cri-o://c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f" gracePeriod=30 Jul 10 18:15:30.790113 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:30.790085 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:31.069942 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.069859 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:31.363533 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.363437 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:31.695729 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.695699 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:15:31.696220 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.696204 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:31.696268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.696222 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:31.696268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.696233 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:31.696268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.696238 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:31.696365 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.696304 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:31.701094 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.701071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.706224 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.706199 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b8bbaf-openshift-default-dockercfg-67c6x\"" Jul 10 18:15:31.713825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.713801 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:15:31.725302 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.725282 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:31.782149 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782110 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9w6b\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782329 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782329 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782329 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.782568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.782509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883374 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883374 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883486 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9w6b\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883513 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883547 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883579 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883799 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.883980 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.883899 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.884402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.884365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.885930 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.885905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.886329 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.886309 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.895626 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.895601 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:31.896109 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:31.896092 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9w6b\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b\") pod \"e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:32.011952 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.011876 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:32.157111 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.157088 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:15:32.159330 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:15:32.159303 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dadb63_1004_4c67_844e_2c92a3cc37af.slice/crio-ee12254bbac7c2444c1e05f35c23b94360fd1647bd86fd7b6eacd74e96b601f7 WatchSource:0}: Error finding container ee12254bbac7c2444c1e05f35c23b94360fd1647bd86fd7b6eacd74e96b601f7: Status 404 returned error can't find the container with id ee12254bbac7c2444c1e05f35c23b94360fd1647bd86fd7b6eacd74e96b601f7 Jul 10 18:15:32.161616 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.161579 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:32.161709 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.161645 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:32.161709 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.161690 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:32.327906 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.327818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerStarted","Data":"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba"} Jul 10 18:15:32.327906 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.327895 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerStarted","Data":"ee12254bbac7c2444c1e05f35c23b94360fd1647bd86fd7b6eacd74e96b601f7"} Jul 10 18:15:32.352799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.352744 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podStartSLOduration=1.352724996 podStartE2EDuration="1.352724996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:15:32.348766164 +0000 UTC m=+2199.409581499" watchObservedRunningTime="2026-07-10 18:15:32.352724996 +0000 UTC m=+2199.413540337" Jul 10 18:15:32.511184 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:32.511155 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:33.013015 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:33.012961 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:33.018552 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:33.018527 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:33.332718 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:33.332632 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:33.333646 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:33.333624 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:34.421453 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:34.421420 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:34.520477 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:34.520451 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:34.611200 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:34.611161 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:34.919402 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:34.919372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:35.387026 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:35.386925 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:35.491260 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:35.491199 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:36.020041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.020022 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:36.123587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123508 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123552 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123588 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123608 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123678 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123704 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123732 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123780 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.123855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmrfn\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn\") pod \"d4724bc0-44d8-4f9f-aafe-c9268631659c\" (UID: \"d4724bc0-44d8-4f9f-aafe-c9268631659c\") " Jul 10 18:15:36.124194 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.123899 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:36.124194 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.124098 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.124194 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.124157 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:36.124194 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.124175 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:36.124406 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.124226 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:15:36.124888 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.124863 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:36.125958 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.125929 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:15:36.126205 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.126183 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn" (OuterVolumeSpecName: "kube-api-access-xmrfn") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "kube-api-access-xmrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:15:36.126461 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.126437 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:15:36.126461 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.126439 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4724bc0-44d8-4f9f-aafe-c9268631659c" (UID: "d4724bc0-44d8-4f9f-aafe-c9268631659c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:15:36.225041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.224982 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmrfn\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-kube-api-access-xmrfn\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225036 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4724bc0-44d8-4f9f-aafe-c9268631659c-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225045 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225264 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225054 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225264 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225063 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225264 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225071 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225264 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225078 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.225264 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.225085 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4724bc0-44d8-4f9f-aafe-c9268631659c-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:15:36.348598 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.348568 2582 generic.go:356] "Generic (PLEG): container finished" podID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerID="c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f" exitCode=0 Jul 10 18:15:36.348764 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.348620 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" Jul 10 18:15:36.348764 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.348648 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerDied","Data":"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f"} Jul 10 18:15:36.348764 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.348709 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh" event={"ID":"d4724bc0-44d8-4f9f-aafe-c9268631659c","Type":"ContainerDied","Data":"23c9d8c28b9f7df73e78000a83d2a8cf9a9a6dfdc6afa43c8ea9bb6f8b88f94f"} Jul 10 18:15:36.348864 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.348774 2582 scope.go:117] "RemoveContainer" containerID="c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f" Jul 10 18:15:36.369921 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.369888 2582 scope.go:117] "RemoveContainer" containerID="b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5" Jul 10 18:15:36.376387 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.376327 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:36.379321 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.379300 2582 scope.go:117] "RemoveContainer" containerID="c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f" Jul 10 18:15:36.379680 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:36.379656 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f\": container with ID starting with c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f not found: ID does not exist" containerID="c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f" Jul 10 18:15:36.379794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.379684 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f"} err="failed to get container status \"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f\": rpc error: code = NotFound desc = could not find container \"c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f\": container with ID starting with c0eeb0c65fa4d89aeb7fecb6c13594bd89e5e74663386677990c33a740077c6f not found: ID does not exist" Jul 10 18:15:36.379794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.379700 2582 scope.go:117] "RemoveContainer" containerID="b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5" Jul 10 18:15:36.380017 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:36.379969 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5\": container with ID starting with b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5 not found: ID does not exist" containerID="b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5" Jul 10 18:15:36.380085 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.380016 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5"} err="failed to get container status \"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5\": rpc error: code = NotFound desc = could not find container \"b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5\": container with ID starting with b046f33cec6160b913a0f2bc3539bad7156ede9dce876dc1eac7bcbadf4317f5 not found: ID does not exist" Jul 10 18:15:36.381078 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:36.381052 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ea3624-openshift-default-7bccb678ff-xzkqh"] Jul 10 18:15:36.393325 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:15:36.393304 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4724bc0_44d8_4f9f_aafe_c9268631659c.slice\": RecentStats: unable to find data in memory cache]" Jul 10 18:15:37.433599 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:37.433568 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" path="/var/lib/kubelet/pods/d4724bc0-44d8-4f9f-aafe-c9268631659c/volumes" Jul 10 18:15:38.948220 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:38.948180 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:39.280054 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.279962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:39.573669 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.573584 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:15:39.574427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574399 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:15:39.574427 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574428 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:15:39.574592 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574444 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:15:39.574592 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574453 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:15:39.574592 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574536 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:15:39.574592 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.574546 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbad365-a07a-4f99-9679-f0acf0ac0a5f" containerName="istio-proxy" Jul 10 18:15:39.579306 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.579285 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.586332 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.586312 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-0af48d-openshift-default-dockercfg-cmvfv\"" Jul 10 18:15:39.598851 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.598819 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:15:39.622066 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.622035 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:39.656385 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656361 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656701 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656701 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pjw\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656701 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.656701 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.656678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758057 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.757982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758252 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758252 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758125 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758390 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758390 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758495 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758495 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758495 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pjw\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758495 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758673 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758678 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758771 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758755 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.758959 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.758935 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.759060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.759044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.760811 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.760791 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.761074 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.761053 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.767011 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.766975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.767198 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.767175 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pjw\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw\") pod \"e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:39.873805 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.873726 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:39.889789 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:39.889767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:40.035797 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.035771 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:15:40.037212 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:15:40.037186 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38af5e3e_8e3e_4da0_bb95_0505bf5b744d.slice/crio-4dc6b37ba0c1abf70a19efd928e167f14d5757a79076475500c80c606fa98fec WatchSource:0}: Error finding container 4dc6b37ba0c1abf70a19efd928e167f14d5757a79076475500c80c606fa98fec: Status 404 returned error can't find the container with id 4dc6b37ba0c1abf70a19efd928e167f14d5757a79076475500c80c606fa98fec Jul 10 18:15:40.039474 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.039441 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:40.039544 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.039506 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:40.039589 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.039552 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:40.368945 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.368906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerStarted","Data":"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790"} Jul 10 18:15:40.369113 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.368965 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerStarted","Data":"4dc6b37ba0c1abf70a19efd928e167f14d5757a79076475500c80c606fa98fec"} Jul 10 18:15:40.392208 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.392156 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podStartSLOduration=1.392143125 podStartE2EDuration="1.392143125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:15:40.390011713 +0000 UTC m=+2207.450827050" watchObservedRunningTime="2026-07-10 18:15:40.392143125 +0000 UTC m=+2207.452958460" Jul 10 18:15:40.890500 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:40.890469 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:41.895840 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:41.895813 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:42.358100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:42.358072 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:42.378886 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:42.378856 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:42.379843 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:42.379821 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:43.932136 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:43.932078 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:46.429171 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:46.429040 2582 generic.go:356] "Generic (PLEG): container finished" podID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerID="863d9fb8e27fef245ed7214e16004e52d995ca17a5c38a07343fb6d0573bbf55" exitCode=137 Jul 10 18:15:46.429171 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:46.429084 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerDied","Data":"863d9fb8e27fef245ed7214e16004e52d995ca17a5c38a07343fb6d0573bbf55"} Jul 10 18:15:46.429171 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:46.429146 2582 scope.go:117] "RemoveContainer" containerID="11c78fcb60c6d55ac15364aadea2b3784bad774bfaf8dd24a249374702ad8716" Jul 10 18:15:46.481570 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:46.481538 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" Jul 10 18:15:46.959322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:46.959273 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:47.435928 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:47.435901 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" Jul 10 18:15:47.436417 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:47.435978 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f" event={"ID":"db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b","Type":"ContainerDied","Data":"376b85c5f60ce61f324f3debde4718d6070909e3b1cb2227ca1d14369371545c"} Jul 10 18:15:47.436417 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:47.436057 2582 scope.go:117] "RemoveContainer" containerID="863d9fb8e27fef245ed7214e16004e52d995ca17a5c38a07343fb6d0573bbf55" Jul 10 18:15:47.469105 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:47.469071 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:47.470941 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:47.470918 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-dfa29f-29728455-qfg6f"] Jul 10 18:15:49.373134 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:49.373093 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 18:15:49.373504 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:49.373159 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 18:15:49.432825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:49.432793 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" path="/var/lib/kubelet/pods/db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b/volumes" Jul 10 18:15:50.457277 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.457241 2582 generic.go:356] "Generic (PLEG): container finished" podID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerID="f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba" exitCode=0 Jul 10 18:15:50.457813 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.457327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerDied","Data":"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba"} Jul 10 18:15:50.457905 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.457886 2582 scope.go:117] "RemoveContainer" containerID="f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba" Jul 10 18:15:50.459108 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.459081 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:50.459195 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.459147 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:50.459195 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:50.459183 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:51.464412 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:51.464372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerStarted","Data":"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458"} Jul 10 18:15:52.012201 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:52.012159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:52.012700 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:52.012664 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 18:15:52.012812 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:52.012753 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 18:15:53.012925 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:53.012882 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 18:15:53.013399 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:53.012947 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 18:15:54.012829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:54.012781 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 18:15:54.013041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:54.012853 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 18:15:55.014303 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:55.014249 2582 patch_prober.go:28] interesting pod/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 10 18:15:55.014773 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:55.014329 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 10 18:15:56.015716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:56.015689 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:56.016129 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:56.015977 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:56.016812 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:56.016794 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:15:57.170873 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:57.170839 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:57.272854 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:57.272820 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:15:58.269304 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.268608 2582 patch_prober.go:28] interesting pod/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41868->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 10 18:15:58.269304 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.268681 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41868->10.133.0.55:15021: read: connection reset by peer" Jul 10 18:15:58.497553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.497471 2582 generic.go:356] "Generic (PLEG): container finished" podID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerID="9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790" exitCode=0 Jul 10 18:15:58.497727 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.497548 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerDied","Data":"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790"} Jul 10 18:15:58.497953 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.497936 2582 scope.go:117] "RemoveContainer" containerID="9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790" Jul 10 18:15:58.499074 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.499046 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:58.499168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.499114 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:58.499168 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:58.499146 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 10 18:15:59.508830 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:59.507826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerStarted","Data":"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e"} Jul 10 18:15:59.890131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:59.890052 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:15:59.890332 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:59.890234 2582 patch_prober.go:28] interesting pod/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 18:15:59.890332 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:15:59.890298 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 18:16:00.012542 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:00.012500 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:16:00.013193 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:00.013144 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" podUID="4a27f047-db08-421b-adfe-bb961e188b81" containerName="cleanup" containerID="cri-o://2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5" gracePeriod=30 Jul 10 18:16:00.891143 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:00.891109 2582 patch_prober.go:28] interesting pod/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 18:16:00.891535 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:00.891160 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 18:16:01.890828 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:01.890787 2582 patch_prober.go:28] interesting pod/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 18:16:01.891073 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:01.890853 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 18:16:02.890188 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:02.890150 2582 patch_prober.go:28] interesting pod/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 18:16:02.890600 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:02.890207 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 18:16:03.372100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:03.372045 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:16:03.893290 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:03.893256 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:16:03.893716 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:03.893485 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:16:03.894166 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:03.894151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:16:05.159712 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:05.159678 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:16:05.260645 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:05.260596 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:16:07.209817 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:07.209773 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:16:30.202757 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.202731 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" Jul 10 18:16:30.664256 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.664222 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a27f047-db08-421b-adfe-bb961e188b81" containerID="2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5" exitCode=137 Jul 10 18:16:30.664435 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.664268 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" Jul 10 18:16:30.664435 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.664272 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" event={"ID":"4a27f047-db08-421b-adfe-bb961e188b81","Type":"ContainerDied","Data":"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5"} Jul 10 18:16:30.664435 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.664325 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2" event={"ID":"4a27f047-db08-421b-adfe-bb961e188b81","Type":"ContainerDied","Data":"fb9cc05b0150504c698a49df681d74dd7ed10375417305774f37ade3c1f87b74"} Jul 10 18:16:30.664435 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.664354 2582 scope.go:117] "RemoveContainer" containerID="2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5" Jul 10 18:16:30.673629 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.673610 2582 scope.go:117] "RemoveContainer" containerID="2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5" Jul 10 18:16:30.673847 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:16:30.673829 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5\": container with ID starting with 2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5 not found: ID does not exist" containerID="2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5" Jul 10 18:16:30.673911 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.673852 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5"} err="failed to get container status \"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5\": rpc error: code = NotFound desc = could not find container \"2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5\": container with ID starting with 2e4b79717e4f50ecaba991decb232ea426a9d31eea033de9e1da07f377d1f4f5 not found: ID does not exist" Jul 10 18:16:30.685373 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.685346 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:16:30.688870 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:30.688850 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728455-ksqw2"] Jul 10 18:16:31.432659 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:31.432624 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a27f047-db08-421b-adfe-bb961e188b81" path="/var/lib/kubelet/pods/4a27f047-db08-421b-adfe-bb961e188b81/volumes" Jul 10 18:16:36.093659 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:16:36.093625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:16.258218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:16.258177 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:16.347779 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:16.347729 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:39.159358 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:39.159321 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:40.071849 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:40.071819 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:47.364905 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.364865 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:18:47.365410 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365400 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365416 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365433 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a27f047-db08-421b-adfe-bb961e188b81" containerName="cleanup" Jul 10 18:18:47.365473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365438 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a27f047-db08-421b-adfe-bb961e188b81" containerName="cleanup" Jul 10 18:18:47.365473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365454 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365473 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365460 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365519 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a27f047-db08-421b-adfe-bb961e188b81" containerName="cleanup" Jul 10 18:18:47.365711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365527 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4724bc0-44d8-4f9f-aafe-c9268631659c" containerName="istio-proxy" Jul 10 18:18:47.365711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365535 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365542 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.365711 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.365548 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:18:47.368562 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.368542 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.372084 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.372060 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"openshift-service-ca.crt\"" Jul 10 18:18:47.372084 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.372071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"sub-test-model-b8bbaf-kserve-self-signed-certs\"" Jul 10 18:18:47.372260 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.372066 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"kube-root-ca.crt\"" Jul 10 18:18:47.372260 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.372071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b8bbaf\"/\"default-dockercfg-sdccd\"" Jul 10 18:18:47.376913 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.376892 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:18:47.392914 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.392886 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:47.431523 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.431665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.431665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.431665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjs9\" (UniqueName: \"kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.431818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.431818 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.431738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533054 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjs9\" (UniqueName: \"kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533230 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533229 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533433 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533302 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533576 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533638 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.533759 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.533738 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.535568 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.535548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.535710 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.535695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.542066 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.542040 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjs9\" (UniqueName: \"kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9\") pod \"sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.678615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.678584 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:47.811481 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:47.811454 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:18:47.813081 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:18:47.813049 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1a25c48_f2cb_43f3_ac49_bea41eb3865d.slice/crio-7a2a3dc065f02a7262e6b22c0666253037c8fd6a59d7864d30863761ccef4d7e WatchSource:0}: Error finding container 7a2a3dc065f02a7262e6b22c0666253037c8fd6a59d7864d30863761ccef4d7e: Status 404 returned error can't find the container with id 7a2a3dc065f02a7262e6b22c0666253037c8fd6a59d7864d30863761ccef4d7e Jul 10 18:18:48.346319 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:48.346270 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" event={"ID":"d1a25c48-f2cb-43f3-ac49-bea41eb3865d","Type":"ContainerStarted","Data":"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996"} Jul 10 18:18:48.346529 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:48.346329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" event={"ID":"d1a25c48-f2cb-43f3-ac49-bea41eb3865d","Type":"ContainerStarted","Data":"7a2a3dc065f02a7262e6b22c0666253037c8fd6a59d7864d30863761ccef4d7e"} Jul 10 18:18:48.363070 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:48.363025 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" podStartSLOduration=1.363009639 podStartE2EDuration="1.363009639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:18:48.361649371 +0000 UTC m=+2395.422464719" watchObservedRunningTime="2026-07-10 18:18:48.363009639 +0000 UTC m=+2395.423824965" Jul 10 18:18:53.514601 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:53.514569 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:18:53.524328 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:53.524304 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-phwsw_a092006b-cbfc-4c4a-81f0-a10014c00c2d/ovn-acl-logging/0.log" Jul 10 18:18:54.647671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:54.647622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:54.704454 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:54.704418 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:18:57.679062 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:57.679027 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:57.679532 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:57.679110 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:57.691559 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:57.691535 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:18:58.409198 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:18:58.409165 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:19:16.848480 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:19:16.848394 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:19:31.490183 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:19:31.490149 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:04.521202 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:04.521164 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:04.622195 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:04.622163 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:15.951601 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:15.951570 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:20:15.977809 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:15.977780 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:16.959657 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:16.959625 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:20.646485 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:20.646444 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:30.972313 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:30.972258 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" podUID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" containerName="main" containerID="cri-o://a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996" gracePeriod=45 Jul 10 18:20:31.233268 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.233212 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:20:31.329214 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329265 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329284 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329347 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329611 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329413 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpjs9\" (UniqueName: \"kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9\") pod \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\" (UID: \"d1a25c48-f2cb-43f3-ac49-bea41eb3865d\") " Jul 10 18:20:31.329611 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329470 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home" (OuterVolumeSpecName: "home") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:31.329611 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329524 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:31.329611 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329554 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:31.329812 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329711 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-model-cache\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.329812 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329729 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tmp-dir\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.329812 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.329742 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-home\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.331671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.331647 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm" (OuterVolumeSpecName: "dshm") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:20:31.331671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.331657 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:20:31.331820 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.331742 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9" (OuterVolumeSpecName: "kube-api-access-dpjs9") pod "d1a25c48-f2cb-43f3-ac49-bea41eb3865d" (UID: "d1a25c48-f2cb-43f3-ac49-bea41eb3865d"). InnerVolumeSpecName "kube-api-access-dpjs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:20:31.430533 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.430502 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-dshm\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.430533 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.430527 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-tls-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.430533 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.430539 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjs9\" (UniqueName: \"kubernetes.io/projected/d1a25c48-f2cb-43f3-ac49-bea41eb3865d-kube-api-access-dpjs9\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:20:31.850703 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.850672 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" containerID="a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996" exitCode=0 Jul 10 18:20:31.850703 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.850704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" event={"ID":"d1a25c48-f2cb-43f3-ac49-bea41eb3865d","Type":"ContainerDied","Data":"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996"} Jul 10 18:20:31.850921 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.850719 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" Jul 10 18:20:31.850921 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.850731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf" event={"ID":"d1a25c48-f2cb-43f3-ac49-bea41eb3865d","Type":"ContainerDied","Data":"7a2a3dc065f02a7262e6b22c0666253037c8fd6a59d7864d30863761ccef4d7e"} Jul 10 18:20:31.850921 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.850748 2582 scope.go:117] "RemoveContainer" containerID="a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996" Jul 10 18:20:31.859377 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.859360 2582 scope.go:117] "RemoveContainer" containerID="a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996" Jul 10 18:20:31.859618 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:20:31.859601 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996\": container with ID starting with a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996 not found: ID does not exist" containerID="a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996" Jul 10 18:20:31.859671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.859626 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996"} err="failed to get container status \"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996\": rpc error: code = NotFound desc = could not find container \"a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996\": container with ID starting with a005d58dffbce2bf9a5b653b13a31f3d001bf8e0b3dfb09bcd07699c342f0996 not found: ID does not exist" Jul 10 18:20:31.865297 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.865276 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:20:31.868381 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:31.868360 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b8bbaf/sub-test-model-b8bbaf-kserve-5c7799f7f7-84hzf"] Jul 10 18:20:33.434397 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:33.434364 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" path="/var/lib/kubelet/pods/d1a25c48-f2cb-43f3-ac49-bea41eb3865d/volumes" Jul 10 18:20:58.097591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:58.097545 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:20:59.429452 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:20:59.429413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:21:05.895543 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:05.895488 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:21:05.996553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:05.996525 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:21:12.163144 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163106 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:21:12.163553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163462 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" containerName="main" Jul 10 18:21:12.163553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163474 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" containerName="main" Jul 10 18:21:12.163553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163492 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:21:12.163553 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163498 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db6db12b-14c6-4e5c-bdcf-9e4cbd6ca61b" containerName="cleanup" Jul 10 18:21:12.163706 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.163564 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a25c48-f2cb-43f3-ac49-bea41eb3865d" containerName="main" Jul 10 18:21:12.166478 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.166457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.169836 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.169811 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"openshift-service-ca.crt\"" Jul 10 18:21:12.169836 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.169828 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"default-dockercfg-jdhgv\"" Jul 10 18:21:12.169973 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.169858 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"rate-test-model-0af48d-kserve-self-signed-certs\"" Jul 10 18:21:12.169973 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.169895 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0af48d\"/\"kube-root-ca.crt\"" Jul 10 18:21:12.176966 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.176947 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:21:12.261689 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.261655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:21:12.311116 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311079 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.311322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.311322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.311322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.311322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311282 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.311322 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.311305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bmb\" (UniqueName: \"kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412516 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bmb\" (UniqueName: \"kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412707 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412680 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.412960 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.413035 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.412975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.413081 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.413048 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.415099 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.415046 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.415217 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.415199 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.420565 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.420543 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bmb\" (UniqueName: \"kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb\") pod \"rate-test-model-0af48d-kserve-c7d49b999-bx9bp\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.476044 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.475976 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:12.611702 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:12.611673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:21:12.614063 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:21:12.614028 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2d906cc_7f98_413a_bd10_229f006eda4b.slice/crio-da6293504451502960a6d2f9e2b18351e7f1a74e7cb0aa609e551458d3df4a00 WatchSource:0}: Error finding container da6293504451502960a6d2f9e2b18351e7f1a74e7cb0aa609e551458d3df4a00: Status 404 returned error can't find the container with id da6293504451502960a6d2f9e2b18351e7f1a74e7cb0aa609e551458d3df4a00 Jul 10 18:21:13.051683 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:13.051604 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" event={"ID":"f2d906cc-7f98-413a-bd10-229f006eda4b","Type":"ContainerStarted","Data":"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067"} Jul 10 18:21:13.051683 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:13.051644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" event={"ID":"f2d906cc-7f98-413a-bd10-229f006eda4b","Type":"ContainerStarted","Data":"da6293504451502960a6d2f9e2b18351e7f1a74e7cb0aa609e551458d3df4a00"} Jul 10 18:21:13.068287 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:13.068234 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" podStartSLOduration=1.068218702 podStartE2EDuration="1.068218702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:21:13.067876098 +0000 UTC m=+2540.128691435" watchObservedRunningTime="2026-07-10 18:21:13.068218702 +0000 UTC m=+2540.129034037" Jul 10 18:21:22.476923 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:22.476881 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:22.476923 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:22.476933 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:22.489471 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:22.489442 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:23.112834 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:23.112807 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:21:44.470258 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:44.470223 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:21:50.186694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:21:50.186659 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:09.125580 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:09.125544 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:10.139855 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:10.139814 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:22:10.160070 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:10.160029 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:25.161121 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.161026 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" podUID="f2d906cc-7f98-413a-bd10-229f006eda4b" containerName="main" containerID="cri-o://d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067" gracePeriod=45 Jul 10 18:22:25.408615 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.408592 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:22:25.420679 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.420609 2582 generic.go:356] "Generic (PLEG): container finished" podID="f2d906cc-7f98-413a-bd10-229f006eda4b" containerID="d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067" exitCode=0 Jul 10 18:22:25.420679 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.420651 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" Jul 10 18:22:25.420825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.420700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" event={"ID":"f2d906cc-7f98-413a-bd10-229f006eda4b","Type":"ContainerDied","Data":"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067"} Jul 10 18:22:25.420825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.420747 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp" event={"ID":"f2d906cc-7f98-413a-bd10-229f006eda4b","Type":"ContainerDied","Data":"da6293504451502960a6d2f9e2b18351e7f1a74e7cb0aa609e551458d3df4a00"} Jul 10 18:22:25.420825 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.420770 2582 scope.go:117] "RemoveContainer" containerID="d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067" Jul 10 18:22:25.431192 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.431173 2582 scope.go:117] "RemoveContainer" containerID="d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067" Jul 10 18:22:25.431513 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:22:25.431493 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067\": container with ID starting with d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067 not found: ID does not exist" containerID="d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067" Jul 10 18:22:25.431618 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.431525 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067"} err="failed to get container status \"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067\": rpc error: code = NotFound desc = could not find container \"d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067\": container with ID starting with d51e8ebfcb4a4ac4f55c13ed1a907c1f5befecb5b4dd51be336f56968a07a067 not found: ID does not exist" Jul 10 18:22:25.516495 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516467 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.516654 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516511 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.516654 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516634 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.516751 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516699 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8bmb\" (UniqueName: \"kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.516794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516748 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.516794 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516772 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:25.516886 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.516807 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home\") pod \"f2d906cc-7f98-413a-bd10-229f006eda4b\" (UID: \"f2d906cc-7f98-413a-bd10-229f006eda4b\") " Jul 10 18:22:25.517088 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.517055 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:25.517226 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.517180 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home" (OuterVolumeSpecName: "home") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:25.517226 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.517198 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-model-cache\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.517226 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.517219 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-tmp-dir\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.519030 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.519009 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm" (OuterVolumeSpecName: "dshm") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:25.519452 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.519431 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 18:22:25.519527 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.519435 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb" (OuterVolumeSpecName: "kube-api-access-g8bmb") pod "f2d906cc-7f98-413a-bd10-229f006eda4b" (UID: "f2d906cc-7f98-413a-bd10-229f006eda4b"). InnerVolumeSpecName "kube-api-access-g8bmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:25.618201 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.618169 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d906cc-7f98-413a-bd10-229f006eda4b-tls-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.618201 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.618194 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-dshm\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.618201 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.618203 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bmb\" (UniqueName: \"kubernetes.io/projected/f2d906cc-7f98-413a-bd10-229f006eda4b-kube-api-access-g8bmb\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.618389 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.618212 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d906cc-7f98-413a-bd10-229f006eda4b-home\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:25.749214 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.749190 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:22:25.752344 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:25.752323 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0af48d/rate-test-model-0af48d-kserve-c7d49b999-bx9bp"] Jul 10 18:22:27.434043 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:27.434014 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d906cc-7f98-413a-bd10-229f006eda4b" path="/var/lib/kubelet/pods/f2d906cc-7f98-413a-bd10-229f006eda4b/volumes" Jul 10 18:22:29.090507 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:29.090463 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:29.419775 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:29.419724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:29.520641 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:29.520604 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:29.620903 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:29.620869 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:29.782791 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:29.782709 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:35.366401 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.366365 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:35.466085 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.466050 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:35.601588 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.601552 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:22:35.601829 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.601800 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" containerID="cri-o://bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458" gracePeriod=30 Jul 10 18:22:35.622636 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.622568 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:35.829935 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.829903 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:35.989969 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:35.989922 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:36.506756 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:36.506724 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:36.609002 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:36.608970 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:36.716766 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:36.716736 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:36.858526 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:36.858455 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:40.857242 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.857219 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:22:40.957693 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957649 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.957882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957703 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.957882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957739 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9w6b\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.957882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957785 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.957882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.957882 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957881 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.958161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957906 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.958161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957935 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.958161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.957971 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert\") pod \"54dadb63-1004-4c67-844e-2c92a3cc37af\" (UID: \"54dadb63-1004-4c67-844e-2c92a3cc37af\") " Jul 10 18:22:40.958161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958055 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:40.958161 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958082 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:40.958423 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958242 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:40.958423 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958394 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:40.958423 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958415 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:40.958591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958431 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:40.958591 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.958410 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:22:40.959295 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.959262 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data" (OuterVolumeSpecName: "istio-data") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:40.960288 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.960261 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token" (OuterVolumeSpecName: "istio-token") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:40.960288 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.960268 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:40.960467 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.960294 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b" (OuterVolumeSpecName: "kube-api-access-k9w6b") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "kube-api-access-k9w6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:40.960467 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:40.960370 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54dadb63-1004-4c67-844e-2c92a3cc37af" (UID: "54dadb63-1004-4c67-844e-2c92a3cc37af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:22:41.059041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.058970 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9w6b\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-kube-api-access-k9w6b\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.059041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.059037 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.059041 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.059047 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.059284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.059055 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.059284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.059063 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54dadb63-1004-4c67-844e-2c92a3cc37af-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.059284 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.059073 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54dadb63-1004-4c67-844e-2c92a3cc37af-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:41.496095 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.496061 2582 generic.go:356] "Generic (PLEG): container finished" podID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerID="bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458" exitCode=0 Jul 10 18:22:41.496276 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.496109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerDied","Data":"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458"} Jul 10 18:22:41.496276 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.496113 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" Jul 10 18:22:41.496276 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.496151 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh" event={"ID":"54dadb63-1004-4c67-844e-2c92a3cc37af","Type":"ContainerDied","Data":"ee12254bbac7c2444c1e05f35c23b94360fd1647bd86fd7b6eacd74e96b601f7"} Jul 10 18:22:41.496276 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.496172 2582 scope.go:117] "RemoveContainer" containerID="bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458" Jul 10 18:22:41.506885 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.506864 2582 scope.go:117] "RemoveContainer" containerID="f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba" Jul 10 18:22:41.514065 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.514037 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:22:41.515941 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.515919 2582 scope.go:117] "RemoveContainer" containerID="bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458" Jul 10 18:22:41.516355 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:22:41.516328 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458\": container with ID starting with bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458 not found: ID does not exist" containerID="bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458" Jul 10 18:22:41.516429 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.516364 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458"} err="failed to get container status \"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458\": rpc error: code = NotFound desc = could not find container \"bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458\": container with ID starting with bef8c026615e69a43dfce8bd74267a871604c3522fb23705b350a976ba391458 not found: ID does not exist" Jul 10 18:22:41.516429 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.516383 2582 scope.go:117] "RemoveContainer" containerID="f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba" Jul 10 18:22:41.516645 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:22:41.516628 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba\": container with ID starting with f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba not found: ID does not exist" containerID="f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba" Jul 10 18:22:41.516694 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.516651 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba"} err="failed to get container status \"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba\": rpc error: code = NotFound desc = could not find container \"f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba\": container with ID starting with f78ae56681755b77dd386de9ce81ec7d53120fa3b403ee7ca6b38aa11745b5ba not found: ID does not exist" Jul 10 18:22:41.517476 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:41.517456 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b8bbaf-openshift-default-785f87fd8c-8pllh"] Jul 10 18:22:42.343293 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.343266 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:42.441517 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.441473 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:42.576172 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.576140 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:22:42.576376 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.576352 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" containerID="cri-o://2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e" gracePeriod=30 Jul 10 18:22:42.599682 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.599610 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:42.827803 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:42.827766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:43.432686 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:43.432656 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" path="/var/lib/kubelet/pods/54dadb63-1004-4c67-844e-2c92a3cc37af/volumes" Jul 10 18:22:47.827939 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.827917 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:22:47.915766 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915733 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.915948 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915777 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.915948 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915804 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4pjw\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.915948 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915850 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.915948 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915874 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.915948 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915907 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.916231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.915984 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.916231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916070 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.916231 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916097 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token\") pod \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\" (UID: \"38af5e3e-8e3e-4da0-bb95-0505bf5b744d\") " Jul 10 18:22:47.916383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916249 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:47.916383 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916262 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 18:22:47.916486 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916415 2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:47.916486 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916435 2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:47.916486 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916463 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:47.916638 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916512 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:47.916728 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.916699 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data" (OuterVolumeSpecName: "istio-data") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:47.918524 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.918493 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 18:22:47.918524 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.918499 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token" (OuterVolumeSpecName: "istio-token") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:47.918799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.918772 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 18:22:47.918799 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.918781 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw" (OuterVolumeSpecName: "kube-api-access-n4pjw") pod "38af5e3e-8e3e-4da0-bb95-0505bf5b744d" (UID: "38af5e3e-8e3e-4da0-bb95-0505bf5b744d"). InnerVolumeSpecName "kube-api-access-n4pjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 18:22:47.962436 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:47.962406 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-sgwmb_e733e6bc-ad67-489e-98fa-7821e493fa92/manager/0.log" Jul 10 18:22:48.017038 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.016987 2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017038 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017040 2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017052 2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017065 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017078 2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017091 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4pjw\" (UniqueName: \"kubernetes.io/projected/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-kube-api-access-n4pjw\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.017250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.017103 2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38af5e3e-8e3e-4da0-bb95-0505bf5b744d-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 10 18:22:48.083100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.083073 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-czk4s_986dc769-d84c-4d51-8a15-a7702bd0afc4/manager/0.log" Jul 10 18:22:48.324047 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.323952 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cj854_a684b129-1905-410d-a262-822ef62bcd0d/server/0.log" Jul 10 18:22:48.439791 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.439764 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8j6s9_26a44ad2-1fed-4d9e-a016-4890b2fb2ea0/manager/0.log" Jul 10 18:22:48.533294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.533260 2582 generic.go:356] "Generic (PLEG): container finished" podID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerID="2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e" exitCode=0 Jul 10 18:22:48.533448 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.533307 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" Jul 10 18:22:48.533448 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.533342 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerDied","Data":"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e"} Jul 10 18:22:48.533448 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.533394 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk" event={"ID":"38af5e3e-8e3e-4da0-bb95-0505bf5b744d","Type":"ContainerDied","Data":"4dc6b37ba0c1abf70a19efd928e167f14d5757a79076475500c80c606fa98fec"} Jul 10 18:22:48.533448 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.533417 2582 scope.go:117] "RemoveContainer" containerID="2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e" Jul 10 18:22:48.542806 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.542791 2582 scope.go:117] "RemoveContainer" containerID="9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790" Jul 10 18:22:48.550908 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.550894 2582 scope.go:117] "RemoveContainer" containerID="2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e" Jul 10 18:22:48.551169 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:22:48.551146 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e\": container with ID starting with 2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e not found: ID does not exist" containerID="2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e" Jul 10 18:22:48.551266 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.551175 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e"} err="failed to get container status \"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e\": rpc error: code = NotFound desc = could not find container \"2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e\": container with ID starting with 2851864c1b9dad743417d6e39aa50e992454d495447c78a7a7d3f227efb1080e not found: ID does not exist" Jul 10 18:22:48.551266 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.551192 2582 scope.go:117] "RemoveContainer" containerID="9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790" Jul 10 18:22:48.551416 ip-10-0-133-236 kubenswrapper[2582]: E0710 18:22:48.551396 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790\": container with ID starting with 9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790 not found: ID does not exist" containerID="9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790" Jul 10 18:22:48.551471 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.551424 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790"} err="failed to get container status \"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790\": rpc error: code = NotFound desc = could not find container \"9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790\": container with ID starting with 9ab3e62b5caf6d72faad85c764591f5e8e46a45064624a4a06807498e09ae790 not found: ID does not exist" Jul 10 18:22:48.554885 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.554862 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:22:48.558432 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.558414 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-0af48d-openshift-default-84d45bf675-rm5vk"] Jul 10 18:22:48.800382 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.800356 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-xfx4b_5f0dca97-484c-4a63-81ee-2959b7867566/copy-manifests/0.log" Jul 10 18:22:48.806810 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:48.806783 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-xfx4b_5f0dca97-484c-4a63-81ee-2959b7867566/manager/0.log" Jul 10 18:22:49.433077 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:49.433050 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" path="/var/lib/kubelet/pods/38af5e3e-8e3e-4da0-bb95-0505bf5b744d/volumes" Jul 10 18:22:50.626250 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:50.626215 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2696x_e1c2ba02-0516-421d-8397-c8287760f5fe/manager/0.log" Jul 10 18:22:50.741156 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:50.741126 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cnrrg_bd3c61da-cc52-4750-8304-ed54560522ca/manager/0.log" Jul 10 18:22:50.965508 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:50.965477 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7t4b9_fd2246d1-f0b9-4817-a935-5dbe2f9b3577/registry-server/0.log" Jul 10 18:22:51.084792 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:51.084762 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ngsmh_96a50536-0a7f-42c3-8237-a5ebeb60cda1/manager/0.log" Jul 10 18:22:51.324848 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:51.324776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-b66sc_da81e2cc-4d37-4c62-bbaf-fdc1531af550/manager/0.log" Jul 10 18:22:51.677745 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:51.677719 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678pqxqt_5268379e-aff0-441a-b984-b806c13d2b65/istio-proxy/0.log" Jul 10 18:22:52.017956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:52.017883 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-prll8_3d008591-1495-44c0-a02e-81631131495a/kube-auth-proxy/0.log" Jul 10 18:22:52.132754 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:52.132730 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/1.log" Jul 10 18:22:52.246363 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:52.246329 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9p28h_0dec4d25-f353-4ec3-94c6-dbf3d606d30d/payload-pre-processing/0.log" Jul 10 18:22:52.924294 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:52.924264 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tf742_9bc7a468-7b19-4195-a042-2fbaaa1cf095/main/0.log" Jul 10 18:22:53.255644 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:53.255566 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jdd8v_e220bf78-46b6-4c28-8340-74b51adfecfa/main/0.log" Jul 10 18:22:57.166600 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166565 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kb8lk/must-gather-vjxxs"] Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166902 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166911 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166924 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166929 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166941 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166946 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166956 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d906cc-7f98-413a-bd10-229f006eda4b" containerName="main" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.166962 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d906cc-7f98-413a-bd10-229f006eda4b" containerName="main" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167032 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d906cc-7f98-413a-bd10-229f006eda4b" containerName="main" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167041 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167047 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167059 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167053 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167425 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167109 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167425 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167116 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38af5e3e-8e3e-4da0-bb95-0505bf5b744d" containerName="istio-proxy" Jul 10 18:22:57.167425 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.167170 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dadb63-1004-4c67-844e-2c92a3cc37af" containerName="istio-proxy" Jul 10 18:22:57.171213 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.171196 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.173446 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.173420 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kb8lk\"/\"default-dockercfg-264mv\"" Jul 10 18:22:57.174354 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.174333 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kb8lk\"/\"openshift-service-ca.crt\"" Jul 10 18:22:57.174461 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.174357 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kb8lk\"/\"kube-root-ca.crt\"" Jul 10 18:22:57.178895 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.178875 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/must-gather-vjxxs"] Jul 10 18:22:57.188750 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.188728 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:57.251140 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.251115 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" limit=1024 Jul 10 18:22:57.294456 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.294434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e19196-ab55-4aba-b24a-fce981d5d04c-must-gather-output\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.294540 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.294475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xd9n\" (UniqueName: \"kubernetes.io/projected/e9e19196-ab55-4aba-b24a-fce981d5d04c-kube-api-access-8xd9n\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.395793 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.395765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e19196-ab55-4aba-b24a-fce981d5d04c-must-gather-output\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.395880 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.395809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xd9n\" (UniqueName: \"kubernetes.io/projected/e9e19196-ab55-4aba-b24a-fce981d5d04c-kube-api-access-8xd9n\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.396155 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.396139 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9e19196-ab55-4aba-b24a-fce981d5d04c-must-gather-output\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.404145 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.404125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xd9n\" (UniqueName: \"kubernetes.io/projected/e9e19196-ab55-4aba-b24a-fce981d5d04c-kube-api-access-8xd9n\") pod \"must-gather-vjxxs\" (UID: \"e9e19196-ab55-4aba-b24a-fce981d5d04c\") " pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.480758 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.480690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" Jul 10 18:22:57.613631 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.613600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/must-gather-vjxxs"] Jul 10 18:22:57.616707 ip-10-0-133-236 kubenswrapper[2582]: W0710 18:22:57.616662 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e19196_ab55_4aba_b24a_fce981d5d04c.slice/crio-b50e23343ea068418e72c1c5d6b4a9172811ae7413fb77bfe41dcf256c3ab6be WatchSource:0}: Error finding container b50e23343ea068418e72c1c5d6b4a9172811ae7413fb77bfe41dcf256c3ab6be: Status 404 returned error can't find the container with id b50e23343ea068418e72c1c5d6b4a9172811ae7413fb77bfe41dcf256c3ab6be Jul 10 18:22:57.618700 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:57.618681 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 18:22:58.590871 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:58.590835 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" event={"ID":"e9e19196-ab55-4aba-b24a-fce981d5d04c","Type":"ContainerStarted","Data":"1eaee952e03c79aae3de8d4a1e2fb5b7da1050bcece6db898594d90be67e3a8b"} Jul 10 18:22:58.591218 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:58.590886 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" event={"ID":"e9e19196-ab55-4aba-b24a-fce981d5d04c","Type":"ContainerStarted","Data":"b50e23343ea068418e72c1c5d6b4a9172811ae7413fb77bfe41dcf256c3ab6be"} Jul 10 18:22:59.597521 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:59.597483 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" event={"ID":"e9e19196-ab55-4aba-b24a-fce981d5d04c","Type":"ContainerStarted","Data":"6869160540d92ce9e456e0d3f305549542f85e061c37163a20ebb6bb8753dfc3"} Jul 10 18:22:59.616540 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:22:59.616488 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kb8lk/must-gather-vjxxs" podStartSLOduration=1.7938452169999999 podStartE2EDuration="2.616472514s" totalImagesPullingTime="822.627297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:22:59.613702048 +0000 UTC m=+2646.674517384" watchObservedRunningTime="2026-07-10 18:22:59.616472514 +0000 UTC m=+2646.677287848" Jul 10 18:23:00.006463 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:00.006437 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gwzp8_542ae712-d1af-4d6d-99ee-60aeb92da71f/global-pull-secret-syncer/0.log" Jul 10 18:23:00.096565 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:00.096532 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lgddx_1bfe168c-9893-4b2e-a197-6943e3f9ec2d/konnectivity-agent/0.log" Jul 10 18:23:00.254665 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:00.254615 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-236.ec2.internal_3a736e251cd3abf42f7ab6349dd18630/haproxy/0.log" Jul 10 18:23:05.161327 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:05.161285 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2696x_e1c2ba02-0516-421d-8397-c8287760f5fe/manager/0.log" Jul 10 18:23:05.191060 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:05.191027 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cnrrg_bd3c61da-cc52-4750-8304-ed54560522ca/manager/0.log" Jul 10 18:23:05.288016 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:05.284728 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7t4b9_fd2246d1-f0b9-4817-a935-5dbe2f9b3577/registry-server/0.log" Jul 10 18:23:05.459635 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:05.459253 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ngsmh_96a50536-0a7f-42c3-8237-a5ebeb60cda1/manager/0.log" Jul 10 18:23:05.549194 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:05.549120 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-b66sc_da81e2cc-4d37-4c62-bbaf-fdc1531af550/manager/0.log" Jul 10 18:23:07.729743 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:07.729706 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2f6c_ba52d2bc-622c-430e-b437-aa91152f4524/node-exporter/0.log" Jul 10 18:23:07.751587 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:07.751503 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2f6c_ba52d2bc-622c-430e-b437-aa91152f4524/kube-rbac-proxy/0.log" Jul 10 18:23:07.777780 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:07.777754 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2f6c_ba52d2bc-622c-430e-b437-aa91152f4524/init-textfile/0.log" Jul 10 18:23:07.965710 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:07.965682 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/prometheus/0.log" Jul 10 18:23:07.996255 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:07.996207 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/config-reloader/0.log" Jul 10 18:23:08.022695 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:08.022658 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/thanos-sidecar/0.log" Jul 10 18:23:08.043267 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:08.043221 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/kube-rbac-proxy-web/0.log" Jul 10 18:23:08.069388 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:08.069359 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/kube-rbac-proxy/0.log" Jul 10 18:23:08.093698 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:08.093670 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/kube-rbac-proxy-thanos/0.log" Jul 10 18:23:08.115701 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:08.115669 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a03920cb-63ee-4314-8fc3-7ca20f0c8254/init-config-reloader/0.log" Jul 10 18:23:10.542671 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.542636 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7"] Jul 10 18:23:10.549012 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.548966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.552480 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.552405 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7"] Jul 10 18:23:10.640843 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.640806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-proc\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.641171 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.641151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-sys\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.641364 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.641347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-lib-modules\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.641471 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.641457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-podres\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.641640 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.641611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-kube-api-access-hppsn\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742589 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-lib-modules\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-podres\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742622 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-kube-api-access-hppsn\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-proc\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742717 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-sys\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742763 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-podres\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742983 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742765 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-lib-modules\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742983 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742827 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-sys\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.742983 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.742831 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-proc\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.750206 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.750180 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppsn\" (UniqueName: \"kubernetes.io/projected/aa1418ba-b7ed-44e9-b948-1354f7e4c62e-kube-api-access-hppsn\") pod \"perf-node-gather-daemonset-xdcz7\" (UID: \"aa1418ba-b7ed-44e9-b948-1354f7e4c62e\") " pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:10.863677 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:10.863569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:11.042158 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:11.040712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7"] Jul 10 18:23:11.687594 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:11.687539 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" event={"ID":"aa1418ba-b7ed-44e9-b948-1354f7e4c62e","Type":"ContainerStarted","Data":"f4d074dd88319b391e7c960b341d7158972dd2278395941b2d66f5453fbcdf3e"} Jul 10 18:23:11.688100 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:11.687638 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" event={"ID":"aa1418ba-b7ed-44e9-b948-1354f7e4c62e","Type":"ContainerStarted","Data":"5e311e4d618562c7078d1c6af599155f8970b01e826475db5c276d3a55536c6d"} Jul 10 18:23:11.688911 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:11.688226 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:11.707057 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:11.706969 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" podStartSLOduration=1.706947161 podStartE2EDuration="1.706947161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:23:11.706900485 +0000 UTC m=+2658.767715833" watchObservedRunningTime="2026-07-10 18:23:11.706947161 +0000 UTC m=+2658.767762496" Jul 10 18:23:12.430804 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:12.430775 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nzwqh_d91d859f-8d57-458c-8d15-2c124ece69c7/dns/0.log" Jul 10 18:23:12.449657 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:12.449632 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nzwqh_d91d859f-8d57-458c-8d15-2c124ece69c7/kube-rbac-proxy/0.log" Jul 10 18:23:12.475817 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:12.475787 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4b7hj_35f62177-db28-40d7-a0c6-a2aafe07c415/dns-node-resolver/0.log" Jul 10 18:23:13.055537 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:13.055502 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b8dfdb869-tdt7n_7096471e-80b4-4b31-8594-88e19c4951da/registry/0.log" Jul 10 18:23:13.095554 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:13.095522 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6xr2w_2c38fcda-c9c4-4bd0-85e6-4ce5294b3150/node-ca/0.log" Jul 10 18:23:14.006116 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.006086 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678pqxqt_5268379e-aff0-441a-b984-b806c13d2b65/istio-proxy/0.log" Jul 10 18:23:14.157266 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.157231 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-prll8_3d008591-1495-44c0-a02e-81631131495a/kube-auth-proxy/0.log" Jul 10 18:23:14.183894 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.183867 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/0.log" Jul 10 18:23:14.224548 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.224515 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/1.log" Jul 10 18:23:14.255132 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.255105 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9p28h_0dec4d25-f353-4ec3-94c6-dbf3d606d30d/payload-pre-processing/0.log" Jul 10 18:23:14.997221 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:14.997187 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tz7hx_88201d50-d8af-4155-85d3-21a6c2c9f47a/serve-healthcheck-canary/0.log" Jul 10 18:23:15.502599 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:15.502574 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fb4r_76af51e2-d96d-4878-af5b-73cc8be7e4d8/kube-rbac-proxy/0.log" Jul 10 18:23:15.523112 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:15.523089 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fb4r_76af51e2-d96d-4878-af5b-73cc8be7e4d8/exporter/0.log" Jul 10 18:23:15.544662 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:15.544623 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fb4r_76af51e2-d96d-4878-af5b-73cc8be7e4d8/extractor/0.log" Jul 10 18:23:17.704566 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:17.704536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kb8lk/perf-node-gather-daemonset-xdcz7" Jul 10 18:23:17.722906 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:17.722879 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-sgwmb_e733e6bc-ad67-489e-98fa-7821e493fa92/manager/0.log" Jul 10 18:23:17.890796 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:17.890763 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-czk4s_986dc769-d84c-4d51-8a15-a7702bd0afc4/manager/0.log" Jul 10 18:23:18.005010 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:18.004917 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cj854_a684b129-1905-410d-a262-822ef62bcd0d/server/0.log" Jul 10 18:23:18.081131 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:18.081096 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8j6s9_26a44ad2-1fed-4d9e-a016-4890b2fb2ea0/manager/0.log" Jul 10 18:23:18.237442 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:18.237412 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-xfx4b_5f0dca97-484c-4a63-81ee-2959b7867566/manager/0.log" Jul 10 18:23:18.256527 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:18.256460 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dfdbc665d-xfx4b_5f0dca97-484c-4a63-81ee-2959b7867566/copy-manifests/0.log" Jul 10 18:23:20.764978 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:20.764942 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678pqxqt_5268379e-aff0-441a-b984-b806c13d2b65/istio-proxy/0.log" Jul 10 18:23:20.891735 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:20.891650 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-prll8_3d008591-1495-44c0-a02e-81631131495a/kube-auth-proxy/0.log" Jul 10 18:23:20.908772 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:20.908725 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/0.log" Jul 10 18:23:20.950806 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:20.950771 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/1.log" Jul 10 18:23:20.971854 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:20.971825 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9p28h_0dec4d25-f353-4ec3-94c6-dbf3d606d30d/payload-pre-processing/0.log" Jul 10 18:23:22.206710 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:22.206669 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678pqxqt_5268379e-aff0-441a-b984-b806c13d2b65/istio-proxy/0.log" Jul 10 18:23:22.328934 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:22.328858 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b544bffd9-prll8_3d008591-1495-44c0-a02e-81631131495a/kube-auth-proxy/0.log" Jul 10 18:23:22.354028 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:22.353941 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/0.log" Jul 10 18:23:22.399321 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:22.396489 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-jhpkj_c3f28aaa-bf3a-4b18-bc39-42f789367991/istio-proxy/1.log" Jul 10 18:23:22.424544 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:22.424519 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-9p28h_0dec4d25-f353-4ec3-94c6-dbf3d606d30d/payload-pre-processing/0.log" Jul 10 18:23:25.235447 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:25.235414 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mzfgx_ba6b12af-bddd-4a83-b719-f11bb49d2eec/kube-storage-version-migrator-operator/1.log" Jul 10 18:23:25.236380 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:25.236356 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mzfgx_ba6b12af-bddd-4a83-b719-f11bb49d2eec/kube-storage-version-migrator-operator/0.log" Jul 10 18:23:26.198956 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.198931 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/kube-multus-additional-cni-plugins/0.log" Jul 10 18:23:26.219015 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.218968 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/egress-router-binary-copy/0.log" Jul 10 18:23:26.239195 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.239169 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/cni-plugins/0.log" Jul 10 18:23:26.258438 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.258415 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/bond-cni-plugin/0.log" Jul 10 18:23:26.278951 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.278928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/routeoverride-cni/0.log" Jul 10 18:23:26.298406 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.298382 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/whereabouts-cni-bincopy/0.log" Jul 10 18:23:26.318952 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.318926 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-72p97_d1a3c326-2280-4cdc-9bb8-1b4fe5a74aa2/whereabouts-cni/0.log" Jul 10 18:23:26.678146 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.678113 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fr257_480f4dd5-3168-4d70-bfd4-630cc225498a/kube-multus/0.log" Jul 10 18:23:26.810722 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.810681 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wxtvv_62f47418-60ee-40ac-83f7-9b62d2a18531/network-metrics-daemon/0.log" Jul 10 18:23:26.830392 ip-10-0-133-236 kubenswrapper[2582]: I0710 18:23:26.830370 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wxtvv_62f47418-60ee-40ac-83f7-9b62d2a18531/kube-rbac-proxy/0.log"