Jul 08 17:33:43.582823 ip-10-0-134-201 systemd[1]: Starting Kubernetes Kubelet... Jul 08 17:33:44.001946 ip-10-0-134-201 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:44.001946 ip-10-0-134-201 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 17:33:44.001946 ip-10-0-134-201 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:44.001946 ip-10-0-134-201 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 17:33:44.001946 ip-10-0-134-201 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:33:44.003657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.003564 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 17:33:44.009313 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009294 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.009313 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009312 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009316 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009319 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009324 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009327 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009330 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009333 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009335 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009338 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009341 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009343 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009346 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009348 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009352 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009355 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009358 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009360 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009363 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009365 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009368 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.009379 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009371 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009373 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009376 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009379 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009382 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009385 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009388 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009390 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009393 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009396 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009400 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009404 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009407 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009410 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009412 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009415 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009418 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009421 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009424 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.009887 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009427 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009429 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009432 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009436 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009439 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009441 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009444 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009447 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009450 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009452 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009455 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009457 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009460 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009463 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009467 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009470 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009480 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009483 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009485 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.010387 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009488 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009490 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009493 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009496 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009498 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009501 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009504 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.009506 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010274 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010281 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010285 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010288 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010293 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010296 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010299 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010301 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010304 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010307 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010309 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010312 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.010864 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010315 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010319 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010322 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010324 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010326 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010329 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010332 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010335 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010338 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010341 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010343 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010354 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010358 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010361 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010363 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010366 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010368 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010371 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010374 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.010377 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.011339 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011097 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011104 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011107 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011110 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011113 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011121 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011124 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011127 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011129 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011132 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011135 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011137 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011140 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011145 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011148 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011151 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011154 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011157 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011160 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.011884 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011163 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011167 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011169 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011172 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011175 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011178 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011181 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011184 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011186 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011189 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011192 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011194 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011197 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011199 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011202 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011205 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011207 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011210 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011212 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011219 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.012389 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011221 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011224 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011227 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011230 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011232 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011235 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011237 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011240 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011243 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011245 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011248 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011250 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011253 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011256 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011259 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011262 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011265 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011268 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011270 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.012892 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011274 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011278 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011281 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011283 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011286 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011288 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011291 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011293 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011295 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011298 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011300 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011303 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011305 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011309 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011311 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011314 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011316 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011319 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011322 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011324 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.013358 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011326 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011329 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011331 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011333 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011336 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011339 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011342 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011346 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011348 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011353 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011357 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011360 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011363 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011366 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011368 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011371 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011373 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011376 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011378 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011380 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.013858 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011383 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011799 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011804 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011807 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011811 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011819 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011823 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011826 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011828 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011831 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011834 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011837 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011839 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011842 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011845 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011847 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011850 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011852 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011855 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011858 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.014343 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011860 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011863 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011866 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011868 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011871 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011873 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011876 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011878 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011880 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011883 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011886 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011888 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011891 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011894 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011896 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011899 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011901 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011904 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011908 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011910 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.014855 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011913 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011916 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011919 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011921 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011924 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011928 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011932 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011935 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011938 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011941 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011944 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011946 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011949 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011952 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011955 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011957 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011960 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011962 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011965 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.015340 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011967 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011970 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011972 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011975 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011977 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011980 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011983 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011985 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011988 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011990 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011995 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.011998 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012001 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012003 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012006 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012008 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012011 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012013 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012016 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012019 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.015831 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012022 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012024 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012027 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012029 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012032 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012034 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012037 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012039 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012042 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012044 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012048 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012051 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012054 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012057 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012060 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012062 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012065 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012068 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012070 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.016322 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012073 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012076 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012149 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012160 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012168 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012172 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012177 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012181 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012186 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012194 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012198 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012200 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012205 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012208 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012211 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012214 2578 flags.go:64] FLAG: --cgroup-root="" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012217 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012220 2578 flags.go:64] FLAG: --client-ca-file="" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012223 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012226 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012230 2578 flags.go:64] FLAG: --cluster-domain="" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012233 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012236 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012239 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 17:33:44.016809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012242 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012246 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012249 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012252 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012256 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012259 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012262 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012265 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012270 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012274 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012278 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012281 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012284 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012287 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012290 2578 flags.go:64] FLAG: --enable-server="true" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012295 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012300 2578 flags.go:64] FLAG: --event-burst="100" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012304 2578 flags.go:64] FLAG: --event-qps="50" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012306 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012309 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012312 2578 flags.go:64] FLAG: --eviction-hard="" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012316 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012319 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012322 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012325 2578 flags.go:64] FLAG: --eviction-soft="" Jul 08 17:33:44.017460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012328 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012331 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012334 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012336 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012339 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012342 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012345 2578 flags.go:64] FLAG: --feature-gates="" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012348 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012351 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012355 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012358 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012361 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012364 2578 flags.go:64] FLAG: --help="false" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012367 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012370 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012374 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012376 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012380 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012383 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012386 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012389 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012392 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012396 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 17:33:44.018115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012399 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012403 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012405 2578 flags.go:64] FLAG: --kube-reserved="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012408 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012411 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012414 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012417 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012420 2578 flags.go:64] FLAG: --lock-file="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012423 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012425 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012428 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012433 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012436 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012439 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012442 2578 flags.go:64] FLAG: --logging-format="text" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012444 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012448 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012451 2578 flags.go:64] FLAG: --manifest-url="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012453 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012458 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012461 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012464 2578 flags.go:64] FLAG: --max-pods="110" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012467 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012470 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012476 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 17:33:44.018716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012479 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012481 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012484 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012487 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012495 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012498 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012502 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012505 2578 flags.go:64] FLAG: --pod-cidr="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012509 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012529 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012532 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012535 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012538 2578 flags.go:64] FLAG: --port="10250" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012541 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012544 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ded3364c0a87121" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012547 2578 flags.go:64] FLAG: --qos-reserved="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012550 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012554 2578 flags.go:64] FLAG: --register-node="true" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012556 2578 flags.go:64] FLAG: --register-with-taints="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012560 2578 flags.go:64] FLAG: --registry-burst="10" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012563 2578 flags.go:64] FLAG: --registry-qps="5" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012566 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012568 2578 flags.go:64] FLAG: --reserved-memory="" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012572 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012575 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 17:33:44.019354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012578 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012581 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012584 2578 flags.go:64] FLAG: --runonce="false" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012586 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012589 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012592 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012595 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012599 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012602 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012605 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012608 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012611 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012614 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012618 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012621 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012624 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012627 2578 flags.go:64] FLAG: --system-cgroups="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012630 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012635 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012638 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012641 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012645 2578 flags.go:64] FLAG: --tls-min-version="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012648 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012650 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012654 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 17:33:44.019989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012656 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012659 2578 flags.go:64] FLAG: --v="2" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012663 2578 flags.go:64] FLAG: --version="false" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012667 2578 flags.go:64] FLAG: --vmodule="" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012671 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.012674 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012792 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012795 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012798 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012801 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012805 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012807 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012809 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012812 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012816 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012819 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012821 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012823 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012826 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012828 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012832 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.020631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012834 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012837 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012839 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012843 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012846 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012848 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012851 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012853 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012856 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012859 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012861 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012864 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012866 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012870 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012874 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012877 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012880 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012882 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012884 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.021147 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012887 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012890 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012892 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012895 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012897 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012900 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012903 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012906 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012908 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012910 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012913 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012916 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012921 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012923 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012926 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012929 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012933 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012936 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012939 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012943 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.021665 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012945 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012948 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012951 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012953 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012970 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012973 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012976 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012979 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012982 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012984 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012987 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012990 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012992 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012995 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.012998 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013000 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013004 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013007 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013010 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.022153 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013013 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013016 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013018 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013021 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013023 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013027 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013029 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013032 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013034 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013037 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013040 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013051 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013054 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013056 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013059 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013061 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013064 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013066 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013069 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.022631 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013071 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013074 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013076 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013079 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013081 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013083 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.013086 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.013683 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.021012 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.021033 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.022075 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.022084 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022159 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022166 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022170 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.023098 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022173 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022177 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022180 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022183 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022185 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022188 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022190 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022193 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022196 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022198 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022201 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022204 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022206 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022209 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022211 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022214 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022216 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022219 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022222 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022224 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.023490 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022227 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022229 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022232 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022235 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022237 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022240 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022250 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022253 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022255 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022258 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022260 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022263 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022265 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022268 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022271 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022275 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022279 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022282 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022285 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.023996 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022288 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022291 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022293 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022296 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022298 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022301 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022304 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022306 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022309 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022311 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022313 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022316 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022319 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022323 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022326 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022329 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022332 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022335 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022338 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.024463 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022340 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022357 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022360 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022363 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022366 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022368 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022371 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022374 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022377 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022379 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022382 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022385 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022387 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022390 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022393 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022395 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022398 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022400 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022404 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.024945 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022407 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022411 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022414 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022416 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022419 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022422 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022425 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022427 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022430 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022432 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022435 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022439 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022441 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022444 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022446 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022455 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022457 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022460 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.025407 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022462 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.022468 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022651 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022657 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022660 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022663 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022666 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022668 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022671 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022673 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022676 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022679 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022682 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022684 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022687 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:33:44.025897 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022689 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022692 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022695 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022697 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022700 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022702 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022705 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022707 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022712 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022716 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022720 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022723 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022726 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022729 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022731 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022741 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022744 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022746 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022749 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022751 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:33:44.026262 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022754 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022756 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022759 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022761 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022764 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022766 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022769 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022772 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022774 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022777 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022779 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022781 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022785 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022788 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022791 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022794 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022797 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022799 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022803 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:33:44.026791 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022806 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022808 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022811 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022814 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022817 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022820 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022823 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022826 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022828 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022836 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022839 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022842 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022844 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022847 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022849 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022852 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022854 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022857 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022860 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022862 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:33:44.027279 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022865 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022867 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022869 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022872 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022874 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022877 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022879 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022882 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022884 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022887 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022889 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022892 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022894 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022896 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022899 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022902 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022905 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022908 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022911 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:33:44.027779 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022913 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022916 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022918 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022926 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022929 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022931 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022934 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:44.022936 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.022941 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.023095 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.025417 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.026938 2578 server.go:1019] "Starting client certificate rotation" Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.027085 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:33:44.028224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.027132 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:33:44.029827 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.029812 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:33:44.033784 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.033753 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:33:44.047358 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.047336 2578 log.go:25] "Validated CRI v1 runtime API" Jul 08 17:33:44.049683 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.049667 2578 log.go:25] "Validated CRI v1 image API" Jul 08 17:33:44.051187 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.051169 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 17:33:44.054360 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.054329 2578 fs.go:135] Filesystem UUIDs: map[473ff09e-c808-4634-85ee-dcfb3ef2eb37:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 84e18c34-6798-46ed-9755-4d60bcf8914c:/dev/nvme0n1p3] Jul 08 17:33:44.054407 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.054361 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 17:33:44.059295 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.059171 2578 manager.go:216] Machine: {Timestamp:2026-07-08 17:33:44.058104892 +0000 UTC m=+0.355748051 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100049 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f69ebc8b7d2a700343a4b2691046 SystemUUID:ec22f69e-bc8b-7d2a-7003-43a4b2691046 BootID:d758dcd4-ce8f-4676-9d22-29d0f7ca563b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:30:99:30:92:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:30:99:30:92:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:dc:84:55:7e:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 17:33:44.059295 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.059284 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 17:33:44.059441 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.059366 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 17:33:44.060555 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.060527 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 17:33:44.060741 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.060556 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-201.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 17:33:44.060911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.060892 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:33:44.061311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.061300 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 17:33:44.061355 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.061315 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 17:33:44.061355 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.061324 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:33:44.061938 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.061924 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:33:44.063153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.063142 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 17:33:44.066146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066134 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:33:44.066274 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066264 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 17:33:44.066309 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066300 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 08 17:33:44.066338 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066317 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 17:33:44.066430 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066420 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 17:33:44.066463 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066443 2578 kubelet.go:398] "Adding apiserver pod source" Jul 08 17:33:44.066502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066468 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 17:33:44.066944 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066931 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:33:44.066997 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.066950 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:33:44.070136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.070111 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 17:33:44.071941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.071926 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 17:33:44.071997 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.071956 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 17:33:44.073976 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.073957 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.073983 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074000 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074006 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074014 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074020 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074028 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074033 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 17:33:44.074038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074039 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 17:33:44.074239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074046 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 17:33:44.074239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074057 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 17:33:44.074239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074063 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 17:33:44.074854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074839 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 17:33:44.074854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.074849 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 17:33:44.078400 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.078387 2578 server.go:1268] "Started kubelet" Jul 08 17:33:44.078759 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.078652 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 17:33:44.078832 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.078792 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 17:33:44.078990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.078808 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 17:33:44.078990 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.078941 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 17:33:44.079355 ip-10-0-134-201 systemd[1]: Started Kubernetes Kubelet. Jul 08 17:33:44.081868 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.078939 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 17:33:44.081967 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.081916 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 17:33:44.082383 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.082358 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-201.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 17:33:44.084260 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.084242 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 08 17:33:44.087510 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.087489 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 17:33:44.089243 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.088154 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-201.ec2.internal.18c060ed24b95ca2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-201.ec2.internal,UID:ip-10-0-134-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-201.ec2.internal,},FirstTimestamp:2026-07-08 17:33:44.078359714 +0000 UTC m=+0.376002872,LastTimestamp:2026-07-08 17:33:44.078359714 +0000 UTC m=+0.376002872,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-201.ec2.internal,}" Jul 08 17:33:44.089402 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.089385 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xr97z" Jul 08 17:33:44.089828 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.089811 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 17:33:44.089900 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.089849 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 17:33:44.089943 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.089895 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.090570 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090549 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 17:33:44.090686 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090585 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 17:33:44.090686 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090688 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 17:33:44.090806 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090714 2578 factory.go:55] Registering systemd factory Jul 08 17:33:44.090806 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090733 2578 factory.go:223] Registration of the systemd container factory successfully Jul 08 17:33:44.090899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090813 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 08 17:33:44.090899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.090819 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 08 17:33:44.091162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091147 2578 factory.go:153] Registering CRI-O factory Jul 08 17:33:44.091227 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091165 2578 factory.go:223] Registration of the crio container factory successfully Jul 08 17:33:44.091227 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091217 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 17:33:44.091305 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091243 2578 factory.go:103] Registering Raw factory Jul 08 17:33:44.091305 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091258 2578 manager.go:1196] Started watching for new ooms in manager Jul 08 17:33:44.091959 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.091946 2578 manager.go:349] Starting recovery of all containers Jul 08 17:33:44.097613 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.097582 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 17:33:44.097718 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.097690 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-201.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 17:33:44.102035 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.101869 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xr97z" Jul 08 17:33:44.102466 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.102446 2578 manager.go:354] Recovery completed Jul 08 17:33:44.107471 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.107446 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.110453 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.110426 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.110561 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.110461 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.110561 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.110474 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.111160 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.111147 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 17:33:44.111160 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.111158 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 17:33:44.111311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.111172 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:33:44.111311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.111208 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.112034 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112018 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.112111 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112044 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.112111 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112053 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.112466 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112452 2578 policy_none.go:49] "None policy: Start" Jul 08 17:33:44.112466 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112468 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 17:33:44.112603 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.112478 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 17:33:44.112863 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.112779 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-201.ec2.internal.18c060ed26a2ec83 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-201.ec2.internal,UID:ip-10-0-134-201.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-201.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-201.ec2.internal,},FirstTimestamp:2026-07-08 17:33:44.110443651 +0000 UTC m=+0.408086812,LastTimestamp:2026-07-08 17:33:44.110443651 +0000 UTC m=+0.408086812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-201.ec2.internal,}" Jul 08 17:33:44.113428 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.113414 2578 policy_none.go:47] "Start" Jul 08 17:33:44.144611 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.144594 2578 manager.go:337] "Starting Device Plugin manager" Jul 08 17:33:44.144714 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.144639 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 17:33:44.144714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.144653 2578 server.go:87] "Starting device plugin registration server" Jul 08 17:33:44.144914 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.144901 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 17:33:44.144974 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.144916 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 17:33:44.145020 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.144987 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 17:33:44.145079 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.145066 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 17:33:44.145079 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.145077 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 17:33:44.145709 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.145690 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 17:33:44.145784 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.145750 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.246019 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.245982 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.247087 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.247069 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.247147 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.247104 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.247147 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.247118 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.247147 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.247145 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.255557 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.255492 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.255557 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.255536 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-201.ec2.internal\": node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.272893 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.272869 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.301950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.301916 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 17:33:44.303282 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.303266 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 17:33:44.303392 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.303303 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 17:33:44.303392 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.303330 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 17:33:44.303392 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.303375 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 17:33:44.305837 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.305811 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:44.373146 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.373107 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.403616 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.403587 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"] Jul 08 17:33:44.403722 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.403680 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.405415 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.405398 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.405530 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.405427 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.405530 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.405441 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.406775 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.406763 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.406904 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.406891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.406946 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.406920 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.407572 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407556 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.407710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407589 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.407710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407604 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.407710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407560 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.407710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407662 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.407710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.407673 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.408788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.408773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.408883 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.408802 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:33:44.409496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.409479 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:33:44.409581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.409509 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:33:44.409581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.409544 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:33:44.436263 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.436244 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-201.ec2.internal\" not found" node="ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.440604 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.440589 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-201.ec2.internal\" not found" node="ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.473831 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.473802 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.493101 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.493077 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.493212 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.493105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.493212 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.493122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0779f971e02155c459b2004f758e0fd6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"0779f971e02155c459b2004f758e0fd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.574677 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.574649 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.593332 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.593396 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.593396 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0779f971e02155c459b2004f758e0fd6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"0779f971e02155c459b2004f758e0fd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.593396 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.593496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0779f971e02155c459b2004f758e0fd6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-201.ec2.internal\" (UID: \"0779f971e02155c459b2004f758e0fd6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.593496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.593451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/029788063e3d0bbdf211c91b22cd3513-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal\" (UID: \"029788063e3d0bbdf211c91b22cd3513\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.676483 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.676446 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.741894 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.741845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.742986 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:44.742968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:44.777412 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.777377 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.881219 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.881121 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:44.987750 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:44.987717 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-201.ec2.internal\" not found" Jul 08 17:33:45.008032 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.008008 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:45.026241 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.026213 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 17:33:45.026392 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.026369 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 17:33:45.026433 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.026385 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 17:33:45.067045 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.067017 2578 apiserver.go:52] "Watching apiserver" Jul 08 17:33:45.090044 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.090013 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" Jul 08 17:33:45.090044 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.090038 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 17:33:45.094908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.094883 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 17:33:45.095258 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.095238 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9","openshift-cluster-node-tuning-operator/tuned-ctjfr","openshift-image-registry/node-ca-nllq2","openshift-multus/multus-additional-cni-plugins-bjrww","openshift-network-operator/iptables-alerter-fgnxs","kube-system/global-pull-secret-syncer-8877n","openshift-dns/node-resolver-9x9pr","openshift-multus/multus-h9rp6","openshift-multus/network-metrics-daemon-g4jl4","openshift-network-diagnostics/network-check-target-zhqrh","openshift-ovn-kubernetes/ovnkube-node-fzxjc","kube-system/konnectivity-agent-pcwk6"] Jul 08 17:33:45.096893 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.096876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.098049 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.098028 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.099680 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.099660 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.100825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.100810 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.101256 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.101240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 17:33:45.101425 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.101408 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.101463 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.101415 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.101891 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.101878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkskc\"" Jul 08 17:33:45.102146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.102134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.103307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.103292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.103370 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.103352 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:45.103730 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.103716 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 17:33:45.104034 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104019 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:33:45.104107 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104094 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" Jul 08 17:33:45.104162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104148 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.104376 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104359 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.104413 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.104505 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104489 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.104614 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104598 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 17:33:45.104893 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.104879 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nmqqq\"" Jul 08 17:33:45.105787 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.105765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.105908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.105800 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7m8tw\"" Jul 08 17:33:45.105908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.105821 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 17:33:45.105908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.105889 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.106069 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.105979 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 17:28:44 +0000 UTC" deadline="2028-01-04 06:00:06.012629673 +0000 UTC" Jul 08 17:33:45.106069 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106020 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13068h26m20.906612776s" Jul 08 17:33:45.106069 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106029 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.106198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106086 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 17:33:45.106198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106092 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.106526 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106506 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.106653 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106518 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jwrcm\"" Jul 08 17:33:45.106754 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106736 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 17:33:45.106824 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106794 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.106824 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106747 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.106972 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.106958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c4cn\"" Jul 08 17:33:45.107055 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.107037 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2svwm\"" Jul 08 17:33:45.107152 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.107121 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.107282 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.107266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.107340 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.107323 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:45.108440 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.108421 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:45.108523 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.108487 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:45.109892 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.109877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.111788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.111159 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 17:33:45.111788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.111180 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:33:45.111788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.111244 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v2gbg\"" Jul 08 17:33:45.111788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.111355 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal"] Jul 08 17:33:45.111788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.111423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.112081 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112045 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 17:33:45.112139 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112091 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 17:33:45.112868 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112845 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 17:33:45.112989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112885 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 17:33:45.112989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112927 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 17:33:45.112989 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.112940 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 17:33:45.113144 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.113114 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fdql8\"" Jul 08 17:33:45.113619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.113606 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 17:33:45.113800 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.113788 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8rs65\"" Jul 08 17:33:45.114226 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.114207 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 17:33:45.116843 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.116823 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:33:45.117020 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.117002 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal"] Jul 08 17:33:45.135110 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.135049 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ln6wr" Jul 08 17:33:45.143231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.143211 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ln6wr" Jul 08 17:33:45.192373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.192352 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 17:33:45.197208 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-systemd\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197292 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssssm\" (UniqueName: \"kubernetes.io/projected/695e5fbc-e134-4402-b44c-2780e00cdb77-kube-api-access-ssssm\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.197292 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-netns\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.197292 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-config\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.197464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.197464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-kubernetes\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxck\" (UniqueName: \"kubernetes.io/projected/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-kube-api-access-6pxck\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-kubelet\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.197464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-bin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-socket-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-modprobe-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-sys\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-tuned\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-host\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.197718 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-kubelet\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6qw\" (UniqueName: \"kubernetes.io/projected/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-kube-api-access-ff6qw\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197748 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-tmp\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pkk\" (UniqueName: \"kubernetes.io/projected/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-kube-api-access-68pkk\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-systemd-units\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-etc-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdbv\" (UniqueName: \"kubernetes.io/projected/3d3406ec-1704-4bb6-9dac-18b5d409c65b-kube-api-access-qvdbv\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-socket-dir-parent\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-lib-modules\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197973 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15225d12-e12f-496b-b822-d7f6f5c666a1-agent-certs\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.197995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15225d12-e12f-496b-b822-d7f6f5c666a1-konnectivity-ca\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66fj\" (UniqueName: \"kubernetes.io/projected/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-kube-api-access-d66fj\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-conf\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-env-overrides\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-netns\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cnibin\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-cnibin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-os-release\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695e5fbc-e134-4402-b44c-2780e00cdb77-tmp-dir\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7qd\" (UniqueName: \"kubernetes.io/projected/71fef467-1e6d-45d5-b3fb-78ede7b5caff-kube-api-access-8v7qd\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-system-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198333 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-device-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-host\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-host-slash\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.198735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-script-lib\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysconfig\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-var-lib-kubelet\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-slash\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-var-lib-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-multus-certs\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-registration-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-iptables-alerter-script\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695e5fbc-e134-4402-b44c-2780e00cdb77-hosts-file\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-node-log\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-bin\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.199642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-os-release\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-cni-binary-copy\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xp4\" (UniqueName: \"kubernetes.io/projected/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kube-api-access-p2xp4\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-run\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-systemd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.198996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-netd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199020 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-multus\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-etc-kubernetes\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199070 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24ln\" (UniqueName: \"kubernetes.io/projected/c0808e76-6d3d-42da-8ba4-b39654a015b3-kube-api-access-p24ln\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-sys-fs\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-hostroot\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-conf-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-k8s-cni-cncf-io\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-log-socket\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-serviceca\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-daemon-config\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-kubelet-config\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-ovn\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.200711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.199614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-dbus\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.300482 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxck\" (UniqueName: \"kubernetes.io/projected/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-kube-api-access-6pxck\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300482 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-kubelet\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-bin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-socket-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-modprobe-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-sys\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300594 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-kubelet\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-tuned\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-bin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-sys\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-host\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-kubelet\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300737 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-modprobe-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300742 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-d\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6qw\" (UniqueName: \"kubernetes.io/projected/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-kube-api-access-ff6qw\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-tmp\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-kubelet\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.300790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pkk\" (UniqueName: \"kubernetes.io/projected/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-kube-api-access-68pkk\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-systemd-units\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-host\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-etc-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-socket-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-etc-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdbv\" (UniqueName: \"kubernetes.io/projected/3d3406ec-1704-4bb6-9dac-18b5d409c65b-kube-api-access-qvdbv\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-systemd-units\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.300993 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-socket-dir-parent\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-lib-modules\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15225d12-e12f-496b-b822-d7f6f5c666a1-agent-certs\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-socket-dir-parent\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.301181 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-lib-modules\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15225d12-e12f-496b-b822-d7f6f5c666a1-konnectivity-ca\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.301282 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:33:45.801240212 +0000 UTC m=+2.098883359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:45.301661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66fj\" (UniqueName: \"kubernetes.io/projected/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-kube-api-access-d66fj\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-conf\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301448 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-env-overrides\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-netns\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cnibin\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-cnibin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-os-release\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695e5fbc-e134-4402-b44c-2780e00cdb77-tmp-dir\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7qd\" (UniqueName: \"kubernetes.io/projected/71fef467-1e6d-45d5-b3fb-78ede7b5caff-kube-api-access-8v7qd\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-system-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-device-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-cnibin\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-host\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-host-slash\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.302714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-host\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-system-cni-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-device-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.301955 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-env-overrides\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/15225d12-e12f-496b-b822-d7f6f5c666a1-konnectivity-ca\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302044 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-os-release\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302088 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-script-lib\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302152 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysctl-conf\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-host-slash\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cnibin\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysconfig\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-var-lib-kubelet\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.303619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-slash\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-var-lib-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-multus-certs\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-registration-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695e5fbc-e134-4402-b44c-2780e00cdb77-tmp-dir\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-iptables-alerter-script\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695e5fbc-e134-4402-b44c-2780e00cdb77-hosts-file\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-node-log\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-bin\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-os-release\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-cni-binary-copy\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-node-log\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-bin\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.304497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-os-release\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695e5fbc-e134-4402-b44c-2780e00cdb77-hosts-file\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xp4\" (UniqueName: \"kubernetes.io/projected/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kube-api-access-p2xp4\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.303046 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-run\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.303105 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:33:45.803088332 +0000 UTC m=+2.100731497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.302094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-netns\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-var-lib-openvswitch\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-sysconfig\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-var-lib-kubelet\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-run\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-systemd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-netd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-multus\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-etc-kubernetes\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24ln\" (UniqueName: \"kubernetes.io/projected/c0808e76-6d3d-42da-8ba4-b39654a015b3-kube-api-access-p24ln\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.305756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-iptables-alerter-script\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-cni-binary-copy\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-sys-fs\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-sys-fs\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-multus-certs\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303520 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-systemd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-registration-dir\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303631 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-cni-binary-copy\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-hostroot\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-conf-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303701 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-system-cni-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303770 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-k8s-cni-cncf-io\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-hostroot\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306323 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-var-lib-cni-multus\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-log-socket\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-etc-kubernetes\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-serviceca\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-log-socket\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-daemon-config\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303914 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-kubelet-config\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-cni-netd\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-ovn\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.303972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-slash\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-dbus\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-systemd\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssssm\" (UniqueName: \"kubernetes.io/projected/695e5fbc-e134-4402-b44c-2780e00cdb77-kube-api-access-ssssm\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-netns\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-config\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.306855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-kubernetes\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-kubernetes\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-host-run-k8s-cni-cncf-io\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-kubelet-config\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/65e29b5a-ba76-41f7-8020-8641a67b6eb5-etc-selinux\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-conf-dir\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-run-ovn\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-dbus\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d3406ec-1704-4bb6-9dac-18b5d409c65b-host-run-netns\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.304938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-systemd\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0808e76-6d3d-42da-8ba4-b39654a015b3-multus-daemon-config\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305333 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-serviceca\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/15225d12-e12f-496b-b822-d7f6f5c666a1-agent-certs\") pod \"konnectivity-agent-pcwk6\" (UID: \"15225d12-e12f-496b-b822-d7f6f5c666a1\") " pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305481 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-tmp\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-script-lib\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.307394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.305891 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d3406ec-1704-4bb6-9dac-18b5d409c65b-ovnkube-config\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.307973 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.306153 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-etc-tuned\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.311834 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.311811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxck\" (UniqueName: \"kubernetes.io/projected/436aa101-5bb0-4d6c-a8c2-8cfeb162746b-kube-api-access-6pxck\") pod \"tuned-ctjfr\" (UID: \"436aa101-5bb0-4d6c-a8c2-8cfeb162746b\") " pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.312721 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.312696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6qw\" (UniqueName: \"kubernetes.io/projected/bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9-kube-api-access-ff6qw\") pod \"multus-additional-cni-plugins-bjrww\" (UID: \"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9\") " pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.312816 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.312743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pkk\" (UniqueName: \"kubernetes.io/projected/3b845ba6-2e3f-48de-a463-8db82ac1d2cf-kube-api-access-68pkk\") pod \"iptables-alerter-fgnxs\" (UID: \"3b845ba6-2e3f-48de-a463-8db82ac1d2cf\") " pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.313437 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.313418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdbv\" (UniqueName: \"kubernetes.io/projected/3d3406ec-1704-4bb6-9dac-18b5d409c65b-kube-api-access-qvdbv\") pod \"ovnkube-node-fzxjc\" (UID: \"3d3406ec-1704-4bb6-9dac-18b5d409c65b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.321923 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.321900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssssm\" (UniqueName: \"kubernetes.io/projected/695e5fbc-e134-4402-b44c-2780e00cdb77-kube-api-access-ssssm\") pod \"node-resolver-9x9pr\" (UID: \"695e5fbc-e134-4402-b44c-2780e00cdb77\") " pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.322931 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.322904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xp4\" (UniqueName: \"kubernetes.io/projected/65e29b5a-ba76-41f7-8020-8641a67b6eb5-kube-api-access-p2xp4\") pod \"aws-ebs-csi-driver-node-6klk9\" (UID: \"65e29b5a-ba76-41f7-8020-8641a67b6eb5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.323018 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.322937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24ln\" (UniqueName: \"kubernetes.io/projected/c0808e76-6d3d-42da-8ba4-b39654a015b3-kube-api-access-p24ln\") pod \"multus-h9rp6\" (UID: \"c0808e76-6d3d-42da-8ba4-b39654a015b3\") " pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.323056 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.323034 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:45.323056 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.323054 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:45.323121 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.323065 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:45.323161 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.323126 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:45.823112885 +0000 UTC m=+2.120756055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:45.323205 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.323161 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66fj\" (UniqueName: \"kubernetes.io/projected/58ad5d1d-145d-40b4-abc7-bb7496a78f9a-kube-api-access-d66fj\") pod \"node-ca-nllq2\" (UID: \"58ad5d1d-145d-40b4-abc7-bb7496a78f9a\") " pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.324092 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.324076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7qd\" (UniqueName: \"kubernetes.io/projected/71fef467-1e6d-45d5-b3fb-78ede7b5caff-kube-api-access-8v7qd\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.329630 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.329576 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0779f971e02155c459b2004f758e0fd6.slice/crio-6a84bd4d1119ea51876744005d9e56b350b8bde1f8219abd5866546d3341a3f3 WatchSource:0}: Error finding container 6a84bd4d1119ea51876744005d9e56b350b8bde1f8219abd5866546d3341a3f3: Status 404 returned error can't find the container with id 6a84bd4d1119ea51876744005d9e56b350b8bde1f8219abd5866546d3341a3f3 Jul 08 17:33:45.336224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.336209 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:33:45.362146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.362122 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:45.425166 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.425067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" Jul 08 17:33:45.434660 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.434590 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e29b5a_ba76_41f7_8020_8641a67b6eb5.slice/crio-044bc308733e6f3952e96a3b1198812a901d091678e42742f1f103b2d40e7274 WatchSource:0}: Error finding container 044bc308733e6f3952e96a3b1198812a901d091678e42742f1f103b2d40e7274: Status 404 returned error can't find the container with id 044bc308733e6f3952e96a3b1198812a901d091678e42742f1f103b2d40e7274 Jul 08 17:33:45.439599 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.439571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" Jul 08 17:33:45.446718 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.446696 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436aa101_5bb0_4d6c_a8c2_8cfeb162746b.slice/crio-ea2a6c336d52cb680e72935155ceb1593a24312babc7e8f295c67a2039703737 WatchSource:0}: Error finding container ea2a6c336d52cb680e72935155ceb1593a24312babc7e8f295c67a2039703737: Status 404 returned error can't find the container with id ea2a6c336d52cb680e72935155ceb1593a24312babc7e8f295c67a2039703737 Jul 08 17:33:45.456004 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.455977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nllq2" Jul 08 17:33:45.463144 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.463121 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ad5d1d_145d_40b4_abc7_bb7496a78f9a.slice/crio-275db9e680404bb332f156b74a4067a9315379f2a1495478986a5571554942fd WatchSource:0}: Error finding container 275db9e680404bb332f156b74a4067a9315379f2a1495478986a5571554942fd: Status 404 returned error can't find the container with id 275db9e680404bb332f156b74a4067a9315379f2a1495478986a5571554942fd Jul 08 17:33:45.468606 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.468576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bjrww" Jul 08 17:33:45.475410 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.475379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fgnxs" Jul 08 17:33:45.475678 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.475656 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3e0e2d_0cad_4849_9a61_3c03d68d1fb9.slice/crio-b1e7c535f2d7927e813c4032ced8dcf1b61eeb5b92ae32ec95f01ea55ddf16ef WatchSource:0}: Error finding container b1e7c535f2d7927e813c4032ced8dcf1b61eeb5b92ae32ec95f01ea55ddf16ef: Status 404 returned error can't find the container with id b1e7c535f2d7927e813c4032ced8dcf1b61eeb5b92ae32ec95f01ea55ddf16ef Jul 08 17:33:45.483011 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.482989 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b845ba6_2e3f_48de_a463_8db82ac1d2cf.slice/crio-0f0fc863772b0ffaed893eb0e09c4a00211063590f84140e20fcdf1a6f57a37f WatchSource:0}: Error finding container 0f0fc863772b0ffaed893eb0e09c4a00211063590f84140e20fcdf1a6f57a37f: Status 404 returned error can't find the container with id 0f0fc863772b0ffaed893eb0e09c4a00211063590f84140e20fcdf1a6f57a37f Jul 08 17:33:45.496781 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.496757 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9x9pr" Jul 08 17:33:45.504607 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.504561 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695e5fbc_e134_4402_b44c_2780e00cdb77.slice/crio-c6b2ac5c2fde645b069af87c6a904eb010de6cd27fffb9fb9c8065f16e3530fa WatchSource:0}: Error finding container c6b2ac5c2fde645b069af87c6a904eb010de6cd27fffb9fb9c8065f16e3530fa: Status 404 returned error can't find the container with id c6b2ac5c2fde645b069af87c6a904eb010de6cd27fffb9fb9c8065f16e3530fa Jul 08 17:33:45.507625 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.507610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9rp6" Jul 08 17:33:45.518399 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.518376 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:45.519129 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.519106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:33:45.523495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.523471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:33:45.526499 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.526479 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3406ec_1704_4bb6_9dac_18b5d409c65b.slice/crio-a6b6e903d7596c04877b91cde52d12c7df2e53e9e6def71ebeae95234f5ffa7d WatchSource:0}: Error finding container a6b6e903d7596c04877b91cde52d12c7df2e53e9e6def71ebeae95234f5ffa7d: Status 404 returned error can't find the container with id a6b6e903d7596c04877b91cde52d12c7df2e53e9e6def71ebeae95234f5ffa7d Jul 08 17:33:45.531062 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.531034 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15225d12_e12f_496b_b822_d7f6f5c666a1.slice/crio-0ce2770e954d4091ecc5c4f525c501c3c10da8816d1cde8a1a2555651110c701 WatchSource:0}: Error finding container 0ce2770e954d4091ecc5c4f525c501c3c10da8816d1cde8a1a2555651110c701: Status 404 returned error can't find the container with id 0ce2770e954d4091ecc5c4f525c501c3c10da8816d1cde8a1a2555651110c701 Jul 08 17:33:45.809014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.808918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:45.809014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.808979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:45.809196 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.809071 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:45.809196 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.809125 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:33:46.809109658 +0000 UTC m=+3.106752805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:45.809196 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.809072 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:45.809306 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.809207 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:33:46.80919066 +0000 UTC m=+3.106833809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:45.909850 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:45.909822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:45.910007 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.909945 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:45.910007 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.909962 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:45.910007 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.909972 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:45.910139 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:45.910024 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:46.910010638 +0000 UTC m=+3.207653799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:45.974395 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:33:45.974358 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029788063e3d0bbdf211c91b22cd3513.slice/crio-fc99c16263f5a21ec3ac1d2d95ad893da67a05d9212161e908715c10180e91f3 WatchSource:0}: Error finding container fc99c16263f5a21ec3ac1d2d95ad893da67a05d9212161e908715c10180e91f3: Status 404 returned error can't find the container with id fc99c16263f5a21ec3ac1d2d95ad893da67a05d9212161e908715c10180e91f3 Jul 08 17:33:46.145108 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.145002 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:28:45 +0000 UTC" deadline="2028-02-24 03:32:46.267500057 +0000 UTC" Jul 08 17:33:46.145108 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.145053 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14289h59m0.122450111s" Jul 08 17:33:46.305728 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.305694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:46.305921 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.305842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:46.306283 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.306256 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:46.306421 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.306389 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:46.313259 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.313207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"029788063e3d0bbdf211c91b22cd3513","Type":"ContainerStarted","Data":"fc99c16263f5a21ec3ac1d2d95ad893da67a05d9212161e908715c10180e91f3"} Jul 08 17:33:46.321852 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.321821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"a6b6e903d7596c04877b91cde52d12c7df2e53e9e6def71ebeae95234f5ffa7d"} Jul 08 17:33:46.324513 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.324485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9rp6" event={"ID":"c0808e76-6d3d-42da-8ba4-b39654a015b3","Type":"ContainerStarted","Data":"03f00594ca5a7217092d3b5ce391cea46f904dfea2d61b7266583e227af8663f"} Jul 08 17:33:46.327102 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.327069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9x9pr" event={"ID":"695e5fbc-e134-4402-b44c-2780e00cdb77","Type":"ContainerStarted","Data":"c6b2ac5c2fde645b069af87c6a904eb010de6cd27fffb9fb9c8065f16e3530fa"} Jul 08 17:33:46.329085 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.329057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fgnxs" event={"ID":"3b845ba6-2e3f-48de-a463-8db82ac1d2cf","Type":"ContainerStarted","Data":"0f0fc863772b0ffaed893eb0e09c4a00211063590f84140e20fcdf1a6f57a37f"} Jul 08 17:33:46.332576 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.332515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerStarted","Data":"b1e7c535f2d7927e813c4032ced8dcf1b61eeb5b92ae32ec95f01ea55ddf16ef"} Jul 08 17:33:46.334611 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.334580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" event={"ID":"65e29b5a-ba76-41f7-8020-8641a67b6eb5","Type":"ContainerStarted","Data":"044bc308733e6f3952e96a3b1198812a901d091678e42742f1f103b2d40e7274"} Jul 08 17:33:46.336732 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.336661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pcwk6" event={"ID":"15225d12-e12f-496b-b822-d7f6f5c666a1","Type":"ContainerStarted","Data":"0ce2770e954d4091ecc5c4f525c501c3c10da8816d1cde8a1a2555651110c701"} Jul 08 17:33:46.338489 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.338430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nllq2" event={"ID":"58ad5d1d-145d-40b4-abc7-bb7496a78f9a","Type":"ContainerStarted","Data":"275db9e680404bb332f156b74a4067a9315379f2a1495478986a5571554942fd"} Jul 08 17:33:46.340283 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.340161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" event={"ID":"436aa101-5bb0-4d6c-a8c2-8cfeb162746b","Type":"ContainerStarted","Data":"ea2a6c336d52cb680e72935155ceb1593a24312babc7e8f295c67a2039703737"} Jul 08 17:33:46.341327 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.341266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" event={"ID":"0779f971e02155c459b2004f758e0fd6","Type":"ContainerStarted","Data":"6a84bd4d1119ea51876744005d9e56b350b8bde1f8219abd5866546d3341a3f3"} Jul 08 17:33:46.369541 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.369305 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:46.817508 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.817468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:46.817729 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.817549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:46.817729 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.817715 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:46.817851 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.817786 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:33:48.817764168 +0000 UTC m=+5.115407342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:46.818230 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.818213 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:46.818307 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.818267 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:33:48.818251604 +0000 UTC m=+5.115894767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:46.918607 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:46.918530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:46.918775 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.918756 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:46.918835 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.918781 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:46.918835 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.918796 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:46.919071 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:46.918865 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:48.918845854 +0000 UTC m=+5.216489019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:47.145775 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:47.145675 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:28:45 +0000 UTC" deadline="2028-03-10 05:56:40.406034123 +0000 UTC" Jul 08 17:33:47.145775 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:47.145713 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14652h22m53.260326154s" Jul 08 17:33:47.304210 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:47.304153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:47.304414 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:47.304305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:48.035010 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.034973 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:33:48.307000 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.306921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:48.307497 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.307055 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:48.307497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.307469 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:48.307634 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.307569 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:48.837799 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.837756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:48.837986 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.837831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:48.837986 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.837962 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:48.838110 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.838027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:33:52.838006307 +0000 UTC m=+9.135649455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:48.838426 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.838407 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:48.838542 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.838461 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:33:52.838446001 +0000 UTC m=+9.136089162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:48.939140 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:48.939094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:48.939346 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.939323 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:48.939434 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.939353 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:48.939434 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.939368 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:48.939434 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:48.939433 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:33:52.939412188 +0000 UTC m=+9.237055359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:49.303952 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:49.303880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:49.304108 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:49.304017 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:50.303879 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:50.303840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:50.304332 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:50.303840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:50.304332 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:50.303976 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:50.304332 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:50.304058 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:51.303748 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:51.303709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:51.303905 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:51.303856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:52.304448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:52.304418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:52.304839 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.304549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:52.304839 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:52.304643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:52.304839 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.304706 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:52.880532 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:52.880484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:52.880698 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:52.880570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:52.880698 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.880635 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:52.880698 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.880654 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:52.880698 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.880696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:34:00.880682921 +0000 UTC m=+17.178326077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:33:52.880884 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.880715 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:34:00.880704564 +0000 UTC m=+17.178347934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:33:52.981741 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:52.981589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:52.981990 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.981778 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:33:52.981990 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.981802 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:33:52.981990 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.981817 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:52.981990 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:52.981882 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:00.981860813 +0000 UTC m=+17.279503980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:33:53.303898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:53.303814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:53.304083 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:53.303978 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:54.305338 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:54.305301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:54.305759 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:54.305418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:54.305928 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:54.305904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:54.306023 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:54.306003 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:55.305128 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:55.304561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:55.305128 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:55.304721 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:56.304753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:56.304486 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:56.305218 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:56.304858 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:56.305300 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:56.305227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:56.305362 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:56.305321 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:57.303773 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:57.303728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:57.303965 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:57.303866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:33:58.304460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:58.304413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:33:58.304938 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:58.304465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:33:58.305214 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:58.304962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:33:58.305298 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:58.305240 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:33:59.303935 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:33:59.303898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:33:59.304123 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:33:59.304051 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:00.304534 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:00.304487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:00.304977 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:00.304487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:00.304977 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.304628 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:00.304977 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.304680 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:00.943038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:00.942985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:00.943238 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:00.943069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:00.943238 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.943213 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:34:00.943350 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.943274 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:34:16.943255056 +0000 UTC m=+33.240898215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:34:00.943723 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.943694 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:34:00.943854 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:00.943763 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:34:16.943740286 +0000 UTC m=+33.241383451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:34:01.044078 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:01.044039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:01.044272 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:01.044190 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:34:01.044272 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:01.044209 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:34:01.044272 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:01.044218 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:34:01.044272 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:01.044265 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:17.044252219 +0000 UTC m=+33.341895377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:34:01.303575 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:01.303482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:01.303792 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:01.303628 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:02.304315 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:02.304280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:02.304791 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:02.304431 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:02.304791 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:02.304476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:02.304791 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:02.304561 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:03.304117 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:03.304082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:03.304304 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:03.304216 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:04.305529 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:04.305467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:04.305994 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:04.305611 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:04.305994 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:04.305663 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:04.305994 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:04.305776 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:05.304184 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:05.304136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:05.304479 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:05.304285 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:06.304196 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:06.304169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:06.304617 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:06.304317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:06.304617 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:06.304366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:06.304617 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:06.304470 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:06.402546 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:06.402492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" event={"ID":"436aa101-5bb0-4d6c-a8c2-8cfeb162746b","Type":"ContainerStarted","Data":"03463356f75ab8fce8e1c1513c83ece8b7ebc090d3b1a444da3dd3b5deee1939"} Jul 08 17:34:06.404715 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:06.404552 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" event={"ID":"0779f971e02155c459b2004f758e0fd6","Type":"ContainerStarted","Data":"29b95abb36e16c5c2ac5d37944359771e0a47bfa77575ed01f419e59f02027da"} Jul 08 17:34:06.426235 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:06.425901 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ctjfr" podStartSLOduration=1.6779209449999999 podStartE2EDuration="22.425881191s" totalImagesPullingTime="20.747960246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:06.425793305 +0000 UTC m=+22.723436473" watchObservedRunningTime="2026-07-08 17:34:06.425881191 +0000 UTC m=+22.723524359" Jul 08 17:34:07.304571 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.304237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:07.305345 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:07.304680 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:07.413336 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.413305 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="33cfed2c2808eb587d5eaa3e80b8ad8ac7d66c1b787cd25a6da946291bcb2d30" exitCode=0 Jul 08 17:34:07.413567 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.413376 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"33cfed2c2808eb587d5eaa3e80b8ad8ac7d66c1b787cd25a6da946291bcb2d30"} Jul 08 17:34:07.414972 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.414869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" event={"ID":"65e29b5a-ba76-41f7-8020-8641a67b6eb5","Type":"ContainerStarted","Data":"2a19256f6d3f9d9dd0f6be7c1b0aaca09918cc9d040f57c2c728d771ba82d94c"} Jul 08 17:34:07.416214 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.416191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pcwk6" event={"ID":"15225d12-e12f-496b-b822-d7f6f5c666a1","Type":"ContainerStarted","Data":"382d453ac0151e03412b5ab53f2c9a1e7ea1a9e3eb23549edf8634617d084556"} Jul 08 17:34:07.417586 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.417566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nllq2" event={"ID":"58ad5d1d-145d-40b4-abc7-bb7496a78f9a","Type":"ContainerStarted","Data":"190caf00c2aa8d7a9d201d354d12c78c87422771dc37ac91142ca1e98657a4d1"} Jul 08 17:34:07.418958 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.418939 2578 generic.go:356] "Generic (PLEG): container finished" podID="029788063e3d0bbdf211c91b22cd3513" containerID="bd3438101c2f8ff7490c73621735f63af9d3b19d96e74e6a29be7f426b28b27e" exitCode=0 Jul 08 17:34:07.419051 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.419002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"029788063e3d0bbdf211c91b22cd3513","Type":"ContainerDied","Data":"bd3438101c2f8ff7490c73621735f63af9d3b19d96e74e6a29be7f426b28b27e"} Jul 08 17:34:07.422559 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"30526aec3dbfa2ddfab67e244b19543d632108a868f1f95cf47f55987e4b6111"} Jul 08 17:34:07.422649 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422598 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"b7f3e7a54a76d78a24e543e78d6bc29d7719b0c03af6e7648782e38fbf03a63b"} Jul 08 17:34:07.422695 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"5e42b62f9e5d8585a6bd69fc12eb4a73a3e90f2301e3e4065576bc89ee1c1cc6"} Jul 08 17:34:07.422695 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"b69916fd9e3d60b80f40f83d32a204ba88b37e7a8cb482dad494884e3af6a486"} Jul 08 17:34:07.422793 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"5f9e6abc4343e01299766876e5d0040b511370366ba84d03e607e4ceb453b7c5"} Jul 08 17:34:07.422793 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.422756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"248939752d11e00bcd03b9fcb1b021fb8b794c29de8ef23c8574078804b63a0b"} Jul 08 17:34:07.423886 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.423867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9rp6" event={"ID":"c0808e76-6d3d-42da-8ba4-b39654a015b3","Type":"ContainerStarted","Data":"15626a80ca0b8a7a7e2513d5973b36f65c03a42e26f2c62202ea39456e7b3ec5"} Jul 08 17:34:07.425163 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.425146 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9x9pr" event={"ID":"695e5fbc-e134-4402-b44c-2780e00cdb77","Type":"ContainerStarted","Data":"4dbc7a79ac58e7d89ef7a8bb0d8b65d6bae0514681ca78fa8eff38ef2cf11dce"} Jul 08 17:34:07.442970 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.442921 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-201.ec2.internal" podStartSLOduration=22.442901567 podStartE2EDuration="22.442901567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:06.441442705 +0000 UTC m=+22.739085874" watchObservedRunningTime="2026-07-08 17:34:07.442901567 +0000 UTC m=+23.740544735" Jul 08 17:34:07.459245 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.459200 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pcwk6" podStartSLOduration=2.766904341 podStartE2EDuration="23.459183174s" totalImagesPullingTime="20.692278833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:07.458773161 +0000 UTC m=+23.756416330" watchObservedRunningTime="2026-07-08 17:34:07.459183174 +0000 UTC m=+23.756826340" Jul 08 17:34:07.489964 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.489211 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nllq2" podStartSLOduration=2.757714381 podStartE2EDuration="23.489191947s" totalImagesPullingTime="20.731477566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:07.487704277 +0000 UTC m=+23.785347445" watchObservedRunningTime="2026-07-08 17:34:07.489191947 +0000 UTC m=+23.786835115" Jul 08 17:34:07.523647 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.523600 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9x9pr" podStartSLOduration=2.804884687 podStartE2EDuration="23.523562817s" totalImagesPullingTime="20.71867813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:07.502421656 +0000 UTC m=+23.800064822" watchObservedRunningTime="2026-07-08 17:34:07.523562817 +0000 UTC m=+23.821205983" Jul 08 17:34:07.523783 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.523691 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h9rp6" podStartSLOduration=2.7479731689999998 podStartE2EDuration="23.523682653s" totalImagesPullingTime="20.775709484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:07.522900213 +0000 UTC m=+23.820543383" watchObservedRunningTime="2026-07-08 17:34:07.523682653 +0000 UTC m=+23.821325820" Jul 08 17:34:07.656051 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.656026 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9x9pr_695e5fbc-e134-4402-b44c-2780e00cdb77/dns-node-resolver/0.log" Jul 08 17:34:07.672960 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.672920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:07.771650 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:07.771573 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:08.293740 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.293695 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:08.303739 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.303699 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:08.303908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.303705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:08.303908 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:08.303831 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:08.304016 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:08.303916 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:08.393171 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.393147 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:08.429681 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.429607 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" event={"ID":"029788063e3d0bbdf211c91b22cd3513","Type":"ContainerStarted","Data":"da86d610f421924599a51e8f6e591bdb7b1a82555e534526994883d5df088ab9"} Jul 08 17:34:08.431833 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.431799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fgnxs" event={"ID":"3b845ba6-2e3f-48de-a463-8db82ac1d2cf","Type":"ContainerStarted","Data":"dc48c920dae1b0e53fee675fdfa25306aa5276af596617491959b5ad3103646e"} Jul 08 17:34:08.445558 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.445495 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-201.ec2.internal" podStartSLOduration=23.445479438 podStartE2EDuration="23.445479438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:08.445220902 +0000 UTC m=+24.742864076" watchObservedRunningTime="2026-07-08 17:34:08.445479438 +0000 UTC m=+24.743122605" Jul 08 17:34:08.460395 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.460343 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fgnxs" podStartSLOduration=3.712160749 podStartE2EDuration="24.460316367s" totalImagesPullingTime="20.748155618s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:08.459888073 +0000 UTC m=+24.757531241" watchObservedRunningTime="2026-07-08 17:34:08.460316367 +0000 UTC m=+24.757959587" Jul 08 17:34:08.500081 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.500053 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 17:34:08.809244 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.809214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:08.830836 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.830808 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nllq2_58ad5d1d-145d-40b4-abc7-bb7496a78f9a/node-ca/0.log" Jul 08 17:34:08.903916 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.903879 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:08.958959 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:08.958926 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:09.172042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.171913 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T17:34:08.500076057Z","UUID":"7bb2425c-8def-4ce7-819e-38fec4809402","Handler":null,"Name":"","Endpoint":""} Jul 08 17:34:09.173671 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.173644 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 17:34:09.173813 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.173691 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 17:34:09.200848 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.200811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:09.301271 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.301236 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:09.304237 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.304212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:09.304363 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:09.304315 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:09.437289 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.437149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" event={"ID":"65e29b5a-ba76-41f7-8020-8641a67b6eb5","Type":"ContainerStarted","Data":"310e8d8b409f533b8f5ebc539c2ef75602df438a3716a3c0cbfaffd3c81a7550"} Jul 08 17:34:09.442060 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.442028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"f907f0852d901253509040386803407af93647cfe5676754a4e1a9a92678b24e"} Jul 08 17:34:09.634829 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.634797 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:09.906160 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:09.906123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:10.154580 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.154546 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:10.252303 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.252217 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:10.304270 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.304242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:10.304441 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.304252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:10.304441 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:10.304371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:10.304561 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:10.304435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:10.447703 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.447668 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" event={"ID":"65e29b5a-ba76-41f7-8020-8641a67b6eb5","Type":"ContainerStarted","Data":"81eda7cefe34e7c9bd97599fa57076228d1164ce33e1b8fd2abd7601b07b6ca7"} Jul 08 17:34:10.470499 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.470448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6klk9" podStartSLOduration=2.7752450079999997 podStartE2EDuration="26.470427202s" totalImagesPullingTime="23.695182194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:10.470087282 +0000 UTC m=+26.767730452" watchObservedRunningTime="2026-07-08 17:34:10.470427202 +0000 UTC m=+26.768070370" Jul 08 17:34:10.972158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:10.972121 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:11.020904 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.020867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:34:11.021446 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.021428 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:34:11.113884 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.113854 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:11.213751 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.213717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:11.304158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.304081 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:11.304321 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:11.304218 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:11.319844 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.319800 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:11.436249 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.436215 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:11.450364 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.449906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:34:11.450794 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:11.450641 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pcwk6" Jul 08 17:34:12.053247 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:12.053213 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:12.303862 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:12.303784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:12.304016 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:12.303793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:12.304016 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:12.303905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:12.304016 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:12.304002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:12.648120 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:12.647992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:12.854646 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:12.854580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:13.250981 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.250896 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:13.304137 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.304111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:13.304258 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:13.304223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:13.398858 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.398823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:13.457256 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.457214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" event={"ID":"3d3406ec-1704-4bb6-9dac-18b5d409c65b","Type":"ContainerStarted","Data":"fb2720bae2e5174df4852ef8ce726ddf63c6570a3df40d928bf3f6d1a56d9459"} Jul 08 17:34:13.457441 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.457425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:13.458974 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.458946 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="963ad563dbd6c2421134c7f8959e5c3bafe6587826e7ef397ec0d694ca08153c" exitCode=0 Jul 08 17:34:13.459104 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.459029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"963ad563dbd6c2421134c7f8959e5c3bafe6587826e7ef397ec0d694ca08153c"} Jul 08 17:34:13.474496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.474468 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:13.497795 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.497747 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" podStartSLOduration=8.731698845 podStartE2EDuration="29.497732874s" totalImagesPullingTime="20.766034029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:13.497204603 +0000 UTC m=+29.794847771" watchObservedRunningTime="2026-07-08 17:34:13.497732874 +0000 UTC m=+29.795376035" Jul 08 17:34:13.501941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.501858 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:13.845426 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:13.845315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:14.305203 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.304994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:14.305355 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.305055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:14.305355 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:14.305272 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:14.305429 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:14.305348 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:14.462047 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.462012 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:14.462220 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.462080 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:14.477894 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.477864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:14.550398 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.550365 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:14.651022 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.650986 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:14.764820 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.764785 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:14.865885 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:14.865825 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:15.248179 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:15.248083 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:15.304325 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:15.304290 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:15.304479 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:15.304401 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:15.445751 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:15.445721 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:15.465957 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:15.465925 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="a8871d495f4a480d2a34a9fe9a640b5ba00eccccf958a8c2ee439a1f99b74fef" exitCode=0 Jul 08 17:34:15.466162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:15.465960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"a8871d495f4a480d2a34a9fe9a640b5ba00eccccf958a8c2ee439a1f99b74fef"} Jul 08 17:34:16.260159 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.260128 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:16.304097 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.304060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:16.304097 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.304093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:16.304309 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.304184 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:16.304309 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.304293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:16.360761 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.360735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:16.655476 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.655445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:16.957036 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.956966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:16.957036 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:16.957015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:16.957186 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.957113 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:34:16.957186 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.957117 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:34:16.957186 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.957168 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs podName:71fef467-1e6d-45d5-b3fb-78ede7b5caff nodeName:}" failed. No retries permitted until 2026-07-08 17:34:48.957150898 +0000 UTC m=+65.254794057 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs") pod "network-metrics-daemon-g4jl4" (UID: "71fef467-1e6d-45d5-b3fb-78ede7b5caff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:34:16.957186 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:16.957181 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret podName:d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f nodeName:}" failed. No retries permitted until 2026-07-08 17:34:48.957175585 +0000 UTC m=+65.254818730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret") pod "global-pull-secret-syncer-8877n" (UID: "d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:34:17.057501 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.057473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:17.057653 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:17.057627 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:34:17.057653 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:17.057645 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:34:17.057653 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:17.057655 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mtfm9 for pod openshift-network-diagnostics/network-check-target-zhqrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:34:17.057746 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:17.057700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9 podName:461ece89-2553-4c64-8af2-80b73de11b77 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:49.057687394 +0000 UTC m=+65.355330539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtfm9" (UniqueName: "kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9") pod "network-check-target-zhqrh" (UID: "461ece89-2553-4c64-8af2-80b73de11b77") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:34:17.261204 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.261124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:17.303674 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.303648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:17.303796 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:17.303742 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:17.324881 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.324853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:17.461702 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.461671 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:17.472347 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.472316 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="b9f2ee563287b0e9ce00aa7c85d49fff60a71f60e9ad7664477fd7a1856163cb" exitCode=0 Jul 08 17:34:17.472467 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:17.472377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"b9f2ee563287b0e9ce00aa7c85d49fff60a71f60e9ad7664477fd7a1856163cb"} Jul 08 17:34:18.048466 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:18.048434 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:18.303976 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:18.303891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:18.304403 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:18.303989 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:18.304403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:18.304045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:18.304403 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:18.304132 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:18.642287 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:18.642260 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:18.844164 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:18.844123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:19.246011 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:19.245979 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:19.303583 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:19.303555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:19.303733 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:19.303694 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:19.455297 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:19.455266 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:19.841173 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:19.841138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:20.303638 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:20.303606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:20.303824 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:20.303607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:20.303824 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:20.303742 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:20.304547 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:20.304235 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:20.454695 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:20.454656 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:20.847299 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:20.847252 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:21.247058 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:21.247017 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:21.304317 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:21.304280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:21.304471 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:21.304431 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:21.449544 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:21.449493 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:21.643382 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:21.643350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:22.303665 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:22.303631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:22.304128 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:22.303681 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:22.304128 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:22.303779 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:22.304128 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:22.303927 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:23.304448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:23.304413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:23.304859 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:23.304539 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:23.487915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:23.487883 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="461cd058c110ae62f3b491ad54f0fa49f12afd5fa021c631262bfb5ee42b5e42" exitCode=0 Jul 08 17:34:23.488065 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:23.487942 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"461cd058c110ae62f3b491ad54f0fa49f12afd5fa021c631262bfb5ee42b5e42"} Jul 08 17:34:24.304591 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.304552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:24.304591 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.304594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:24.305078 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:24.304664 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:24.305078 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:24.304773 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:24.492632 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.492603 2578 generic.go:356] "Generic (PLEG): container finished" podID="bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9" containerID="4f8b1591e74dc2c3c2a84ec6bfd078a9970cbfb6f1d294166b578d1b740daae6" exitCode=0 Jul 08 17:34:24.492787 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.492639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerDied","Data":"4f8b1591e74dc2c3c2a84ec6bfd078a9970cbfb6f1d294166b578d1b740daae6"} Jul 08 17:34:24.810399 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.810350 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:24.874346 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:24.874307 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:25.303738 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:25.303667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:25.303883 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:25.303804 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:25.498276 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:25.498244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bjrww" event={"ID":"bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9","Type":"ContainerStarted","Data":"87c081893b90fe6a0ba9ef0bcc24603b905c8300482c353a625f648cba78a887"} Jul 08 17:34:25.521231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:25.521185 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bjrww" podStartSLOduration=7.285304978 podStartE2EDuration="41.521171512s" totalImagesPullingTime="34.235866534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:25.519993294 +0000 UTC m=+41.817636462" watchObservedRunningTime="2026-07-08 17:34:25.521171512 +0000 UTC m=+41.818814658" Jul 08 17:34:26.304303 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:26.304273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:26.304481 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:26.304388 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:26.304481 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:26.304451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:26.304598 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:26.304578 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:27.095433 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.095392 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:27.198779 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.198741 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:27.304381 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.304346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:27.304569 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:27.304454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:27.367924 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.367892 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:27.884177 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.884118 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:27.987184 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:27.987138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:28.156743 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:28.156661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:28.303790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:28.303750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:28.303790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:28.303792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:28.303988 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:28.303889 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:28.304029 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:28.303994 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:28.757031 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:28.756991 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:28.950006 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:28.949977 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:29.211134 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:29.211103 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:29.304213 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:29.304186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:29.304362 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:29.304298 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:30.304088 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:30.304061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:30.304088 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:30.304091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:30.304578 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:30.304167 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:30.304578 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:30.304235 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:31.015375 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.015340 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:31.176836 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.176804 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:31.268679 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.268602 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:31.303999 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.303973 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:31.304130 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:31.304070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:31.370194 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.370169 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:31.470034 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:31.470005 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:32.304031 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.303993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:32.304209 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.304009 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:32.304209 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:32.304117 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:32.304652 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:32.304463 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:32.630209 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.630022 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhqrh"] Jul 08 17:34:32.630375 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.630284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:32.630445 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:32.630377 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:32.633922 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.633899 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8877n"] Jul 08 17:34:32.634048 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.633994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:32.634131 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:32.634106 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:32.634637 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.634617 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g4jl4"] Jul 08 17:34:32.634913 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:32.634714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:32.634913 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:32.634795 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:34.305002 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.304972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:34.305439 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:34.305059 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:34.305439 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.305143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:34.305439 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.305224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:34.305439 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:34.305266 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:34.305439 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:34.305284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:34.366260 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.366229 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:34.465723 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.465687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:34.532420 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.532389 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:34.609420 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.609388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:34.667193 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.667164 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:34.733714 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:34.733684 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:36.304128 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.304084 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:36.304128 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.304128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:36.304625 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.304085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:36.304625 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:36.304212 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhqrh" podUID="461ece89-2553-4c64-8af2-80b73de11b77" Jul 08 17:34:36.304625 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:36.304309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g4jl4" podUID="71fef467-1e6d-45d5-b3fb-78ede7b5caff" Jul 08 17:34:36.304625 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:36.304405 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8877n" podUID="d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f" Jul 08 17:34:36.527898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.527867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:36.983797 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.983770 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-201.ec2.internal" event="NodeReady" Jul 08 17:34:36.983956 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:36.983915 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 17:34:37.059621 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.059553 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xxntr"] Jul 08 17:34:37.074890 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.074858 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-679nq"] Jul 08 17:34:37.075050 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.074992 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.077593 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.077567 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9pdx\"" Jul 08 17:34:37.077711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.077591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 17:34:37.077711 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.077591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 17:34:37.077944 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.077929 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 17:34:37.078796 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.078769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 17:34:37.094988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.094908 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-679nq"] Jul 08 17:34:37.095146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.095002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.095146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.095014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xxntr"] Jul 08 17:34:37.095146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.095042 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gv4qs"] Jul 08 17:34:37.097809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.097784 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 17:34:37.097930 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.097786 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 17:34:37.097930 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.097841 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wnq9l\"" Jul 08 17:34:37.097930 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.097791 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 17:34:37.108799 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gv4qs"] Jul 08 17:34:37.108917 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.108954 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cec3255-c25a-4c5e-ae32-65e5f5126624-crio-socket\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.108997 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108956 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.108997 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cec3255-c25a-4c5e-ae32-65e5f5126624-data-volume\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.109065 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.108994 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cec3255-c25a-4c5e-ae32-65e5f5126624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.109105 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.109062 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b02a4077-3343-4838-b39a-04e554934119-cert\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.109105 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.109094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zjs\" (UniqueName: \"kubernetes.io/projected/b02a4077-3343-4838-b39a-04e554934119-kube-api-access-82zjs\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.109181 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.109130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb996\" (UniqueName: \"kubernetes.io/projected/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-api-access-tb996\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.111213 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.111193 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 17:34:37.111329 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.111224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 17:34:37.111329 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.111225 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p74p\"" Jul 08 17:34:37.210002 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.209958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b02a4077-3343-4838-b39a-04e554934119-cert\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.210198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zjs\" (UniqueName: \"kubernetes.io/projected/b02a4077-3343-4838-b39a-04e554934119-kube-api-access-82zjs\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.210198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb996\" (UniqueName: \"kubernetes.io/projected/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-api-access-tb996\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cec3255-c25a-4c5e-ae32-65e5f5126624-crio-socket\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cec3255-c25a-4c5e-ae32-65e5f5126624-data-volume\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cec3255-c25a-4c5e-ae32-65e5f5126624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210325 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6cec3255-c25a-4c5e-ae32-65e5f5126624-crio-socket\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7235a83b-fd41-46fe-a258-557f127f8b08-config-volume\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.210436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7235a83b-fd41-46fe-a258-557f127f8b08-metrics-tls\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.210645 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7235a83b-fd41-46fe-a258-557f127f8b08-tmp-dir\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.210645 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cec3255-c25a-4c5e-ae32-65e5f5126624-data-volume\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.210645 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lffn\" (UniqueName: \"kubernetes.io/projected/7235a83b-fd41-46fe-a258-557f127f8b08-kube-api-access-2lffn\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.210781 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.210755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.214241 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.214216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b02a4077-3343-4838-b39a-04e554934119-cert\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.214344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.214240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6cec3255-c25a-4c5e-ae32-65e5f5126624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.219661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.219638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb996\" (UniqueName: \"kubernetes.io/projected/6cec3255-c25a-4c5e-ae32-65e5f5126624-kube-api-access-tb996\") pod \"insights-runtime-extractor-xxntr\" (UID: \"6cec3255-c25a-4c5e-ae32-65e5f5126624\") " pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.220142 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.220117 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zjs\" (UniqueName: \"kubernetes.io/projected/b02a4077-3343-4838-b39a-04e554934119-kube-api-access-82zjs\") pod \"ingress-canary-679nq\" (UID: \"b02a4077-3343-4838-b39a-04e554934119\") " pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.230655 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.230631 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:37.309338 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.309303 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:37.311369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7235a83b-fd41-46fe-a258-557f127f8b08-config-volume\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.311369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7235a83b-fd41-46fe-a258-557f127f8b08-metrics-tls\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.311369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7235a83b-fd41-46fe-a258-557f127f8b08-tmp-dir\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.311545 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lffn\" (UniqueName: \"kubernetes.io/projected/7235a83b-fd41-46fe-a258-557f127f8b08-kube-api-access-2lffn\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.311705 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7235a83b-fd41-46fe-a258-557f127f8b08-tmp-dir\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.311911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.311895 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7235a83b-fd41-46fe-a258-557f127f8b08-config-volume\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.313809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.313789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7235a83b-fd41-46fe-a258-557f127f8b08-metrics-tls\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.320280 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.320254 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lffn\" (UniqueName: \"kubernetes.io/projected/7235a83b-fd41-46fe-a258-557f127f8b08-kube-api-access-2lffn\") pod \"dns-default-gv4qs\" (UID: \"7235a83b-fd41-46fe-a258-557f127f8b08\") " pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.386196 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.386156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xxntr" Jul 08 17:34:37.405388 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.405359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-679nq" Jul 08 17:34:37.419408 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.419387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:37.635415 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.635199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xxntr"] Jul 08 17:34:37.636431 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.636411 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-679nq"] Jul 08 17:34:37.653367 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:37.653338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gv4qs"] Jul 08 17:34:37.662739 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:37.662712 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7235a83b_fd41_46fe_a258_557f127f8b08.slice/crio-eb94c584efe8b37494f238d8c50941f417cf235666fe111967bf389e8bbda9c9 WatchSource:0}: Error finding container eb94c584efe8b37494f238d8c50941f417cf235666fe111967bf389e8bbda9c9: Status 404 returned error can't find the container with id eb94c584efe8b37494f238d8c50941f417cf235666fe111967bf389e8bbda9c9 Jul 08 17:34:38.308907 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.308876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:38.309308 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.309293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:38.309876 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.309679 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:38.313728 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.313692 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktdt9\"" Jul 08 17:34:38.313823 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.313741 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jhm\"" Jul 08 17:34:38.313971 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.313955 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 17:34:38.314136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.314121 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 17:34:38.314301 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.314286 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:34:38.314467 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.314454 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:34:38.531484 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.531449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gv4qs" event={"ID":"7235a83b-fd41-46fe-a258-557f127f8b08","Type":"ContainerStarted","Data":"eb94c584efe8b37494f238d8c50941f417cf235666fe111967bf389e8bbda9c9"} Jul 08 17:34:38.533581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.533550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-679nq" event={"ID":"b02a4077-3343-4838-b39a-04e554934119","Type":"ContainerStarted","Data":"e65a14fdf1298511ede6a85471620ff9155f90b41b428628be04265aab3b2593"} Jul 08 17:34:38.535104 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:38.535048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxntr" event={"ID":"6cec3255-c25a-4c5e-ae32-65e5f5126624","Type":"ContainerStarted","Data":"077e1e0965ae796da69653353c54765b82a30c5e14972c2da38f2b17e313364c"} Jul 08 17:34:41.207206 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:41.207124 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:41.270442 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:41.270409 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:42.546764 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.546722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxntr" event={"ID":"6cec3255-c25a-4c5e-ae32-65e5f5126624","Type":"ContainerStarted","Data":"46a04e732726fb541fb7384ec11ebe6251d72c63cbdfbcb24d5afe77ce6f3de0"} Jul 08 17:34:42.548704 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.548675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gv4qs" event={"ID":"7235a83b-fd41-46fe-a258-557f127f8b08","Type":"ContainerStarted","Data":"a4b6c23a3e3f904483ebe7c72749a2d835ac54acd24524c86fcf48691a23151a"} Jul 08 17:34:42.548829 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.548726 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gv4qs" event={"ID":"7235a83b-fd41-46fe-a258-557f127f8b08","Type":"ContainerStarted","Data":"55f1b4c0e0deb0170833ac6f75123bbb53243b1b7112b6e499cce772222eea0f"} Jul 08 17:34:42.548899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.548826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:42.571017 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.570954 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gv4qs" podStartSLOduration=1.182235827 podStartE2EDuration="5.570933441s" totalImagesPullingTime="4.388697614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:42.570012827 +0000 UTC m=+58.867655993" watchObservedRunningTime="2026-07-08 17:34:42.570933441 +0000 UTC m=+58.868576607" Jul 08 17:34:42.639334 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.639295 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-946gp"] Jul 08 17:34:42.641865 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.641845 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.644254 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644230 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 17:34:42.644254 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644251 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 17:34:42.644410 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644259 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2mhnl\"" Jul 08 17:34:42.644697 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644662 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 17:34:42.644815 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644731 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 17:34:42.644815 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.644803 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 17:34:42.650360 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.650338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-946gp"] Jul 08 17:34:42.757916 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.757868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.757916 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.757911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.758158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.757941 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.758158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.758003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2t6\" (UniqueName: \"kubernetes.io/projected/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-kube-api-access-qw2t6\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.859289 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.859244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.859456 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.859302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.859456 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.859329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.859456 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.859369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2t6\" (UniqueName: \"kubernetes.io/projected/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-kube-api-access-qw2t6\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.860157 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.860130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.863032 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.863012 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.863141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.863088 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.868903 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.868882 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2t6\" (UniqueName: \"kubernetes.io/projected/b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c-kube-api-access-qw2t6\") pod \"prometheus-operator-567f7b8b7c-946gp\" (UID: \"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:42.952319 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:42.952281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" Jul 08 17:34:43.160376 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.160343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-946gp"] Jul 08 17:34:43.163361 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:43.163335 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05fcc7a_9ba8_4f3d_8404_3ead2dcded3c.slice/crio-29c0b5539534caf79c9288fa064dc85ccb169ac87fd227c91942e370f3e01cb1 WatchSource:0}: Error finding container 29c0b5539534caf79c9288fa064dc85ccb169ac87fd227c91942e370f3e01cb1: Status 404 returned error can't find the container with id 29c0b5539534caf79c9288fa064dc85ccb169ac87fd227c91942e370f3e01cb1 Jul 08 17:34:43.552020 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.551936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" event={"ID":"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c","Type":"ContainerStarted","Data":"29c0b5539534caf79c9288fa064dc85ccb169ac87fd227c91942e370f3e01cb1"} Jul 08 17:34:43.553283 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.553258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-679nq" event={"ID":"b02a4077-3343-4838-b39a-04e554934119","Type":"ContainerStarted","Data":"6060d5dd89c0dcee76fafc5d50993c17d200848b59efc1d5b33bc33865a01df1"} Jul 08 17:34:43.569596 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.569543 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-679nq" podStartSLOduration=1.181570094 podStartE2EDuration="6.569502894s" totalImagesPullingTime="5.3879328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:43.569457987 +0000 UTC m=+59.867101155" watchObservedRunningTime="2026-07-08 17:34:43.569502894 +0000 UTC m=+59.867146062" Jul 08 17:34:43.712645 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.712609 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:43.773206 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:43.773168 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:44.557710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:44.557684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" event={"ID":"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c","Type":"ContainerStarted","Data":"85bef297a7012c2a0c5b855b56b1fc1714d85529ed52c391b73d2f2e78e1c8d8"} Jul 08 17:34:44.671168 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:44.671130 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:44.773557 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:44.773458 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:44.848875 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:44.848844 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:45.562687 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:45.562605 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxntr" event={"ID":"6cec3255-c25a-4c5e-ae32-65e5f5126624","Type":"ContainerStarted","Data":"73d27c5821a96267139efcd5244a863f073f43f7b19a65435352728b54b9c0a4"} Jul 08 17:34:45.565777 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:45.564916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" event={"ID":"b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c","Type":"ContainerStarted","Data":"0c4f05363b06752273cf08d242aec9df0b772cd0b0db106bc797c13ef583cf21"} Jul 08 17:34:45.582451 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:45.582404 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-946gp" podStartSLOduration=2.2481854119999998 podStartE2EDuration="3.582388432s" totalImagesPullingTime="1.33420302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:45.581401866 +0000 UTC m=+61.879045085" watchObservedRunningTime="2026-07-08 17:34:45.582388432 +0000 UTC m=+61.880031598" Jul 08 17:34:46.484292 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:46.484264 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxjc" Jul 08 17:34:46.652692 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:46.652642 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:46.748579 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:46.748481 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:46.848659 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:46.848629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:46.988963 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:46.988921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.059927 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.059850 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6zpjm"] Jul 08 17:34:47.062687 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.062664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.068059 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.068035 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 17:34:47.068907 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.068877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 17:34:47.068907 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.068897 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 17:34:47.069470 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.069451 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-glrk5\"" Jul 08 17:34:47.093107 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-metrics-client-ca\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093196 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093122 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8gg\" (UniqueName: \"kubernetes.io/projected/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-kube-api-access-5r8gg\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093196 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-wtmp\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093287 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-sys\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093287 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-root\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093287 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.093403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.093386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-textfile\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.123431 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.123395 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.194680 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8gg\" (UniqueName: \"kubernetes.io/projected/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-kube-api-access-5r8gg\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.194814 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-wtmp\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.194814 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194747 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-sys\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.194814 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-root\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.194814 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-sys\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-wtmp\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194866 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-root\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:47.194973 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.194986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-textfile\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195014 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.195010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-metrics-client-ca\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195283 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:47.195042 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls podName:a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:47.695023866 +0000 UTC m=+63.992667032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls") pod "node-exporter-6zpjm" (UID: "a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8") : secret "node-exporter-tls" not found Jul 08 17:34:47.195330 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.195312 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-textfile\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195485 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.195467 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.195591 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.195487 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-metrics-client-ca\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.197149 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.197134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.204158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.204137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8gg\" (UniqueName: \"kubernetes.io/projected/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-kube-api-access-5r8gg\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.285137 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.285097 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.391462 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.391429 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.481988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.481923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.584854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.584823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.681818 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.681747 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.698837 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.698815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.701237 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.701203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8-node-exporter-tls\") pod \"node-exporter-6zpjm\" (UID: \"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8\") " pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.828852 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.828798 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:47.971403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:47.971309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6zpjm" Jul 08 17:34:47.980927 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:47.980898 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a71e8d_af6c_4dd9_8b57_d15e7330a1d8.slice/crio-a913c898c8679fcb09473b8e2bf7d8d9d7dbdd7e24cb02cce78c5a7ca0d61285 WatchSource:0}: Error finding container a913c898c8679fcb09473b8e2bf7d8d9d7dbdd7e24cb02cce78c5a7ca0d61285: Status 404 returned error can't find the container with id a913c898c8679fcb09473b8e2bf7d8d9d7dbdd7e24cb02cce78c5a7ca0d61285 Jul 08 17:34:48.574506 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:48.574466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zpjm" event={"ID":"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8","Type":"ContainerStarted","Data":"a913c898c8679fcb09473b8e2bf7d8d9d7dbdd7e24cb02cce78c5a7ca0d61285"} Jul 08 17:34:49.011025 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.010650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:49.011025 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.010777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:49.013286 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.013263 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:34:49.013996 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.013961 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:34:49.024624 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.024571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71fef467-1e6d-45d5-b3fb-78ede7b5caff-metrics-certs\") pod \"network-metrics-daemon-g4jl4\" (UID: \"71fef467-1e6d-45d5-b3fb-78ede7b5caff\") " pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:49.025783 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.025701 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:49.026207 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.026163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f-original-pull-secret\") pod \"global-pull-secret-syncer-8877n\" (UID: \"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f\") " pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:49.083706 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.083664 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:49.111308 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.111261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:49.114251 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.114221 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 17:34:49.124661 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.124436 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 17:34:49.135806 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.135777 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfm9\" (UniqueName: \"kubernetes.io/projected/461ece89-2553-4c64-8af2-80b73de11b77-kube-api-access-mtfm9\") pod \"network-check-target-zhqrh\" (UID: \"461ece89-2553-4c64-8af2-80b73de11b77\") " pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:49.135953 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.135888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8877n" Jul 08 17:34:49.145369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.145345 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s4jhm\"" Jul 08 17:34:49.153763 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.153735 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g4jl4" Jul 08 17:34:49.351560 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.351500 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8877n"] Jul 08 17:34:49.355255 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:49.355221 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1921cb0_505b_49ed_b0bc_0c0b0fce8d5f.slice/crio-005a56651d3a8aea2a40bb2f0deb3f20e7a7e28cd0c47a49ce8ece8543598b56 WatchSource:0}: Error finding container 005a56651d3a8aea2a40bb2f0deb3f20e7a7e28cd0c47a49ce8ece8543598b56: Status 404 returned error can't find the container with id 005a56651d3a8aea2a40bb2f0deb3f20e7a7e28cd0c47a49ce8ece8543598b56 Jul 08 17:34:49.369870 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.369841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g4jl4"] Jul 08 17:34:49.374023 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:49.373994 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fef467_1e6d_45d5_b3fb_78ede7b5caff.slice/crio-c9070a604ccf7b038219fd7985d5da7c59acd6a51afcc605d69624dc9c31fecf WatchSource:0}: Error finding container c9070a604ccf7b038219fd7985d5da7c59acd6a51afcc605d69624dc9c31fecf: Status 404 returned error can't find the container with id c9070a604ccf7b038219fd7985d5da7c59acd6a51afcc605d69624dc9c31fecf Jul 08 17:34:49.427937 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.427911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktdt9\"" Jul 08 17:34:49.436171 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.436132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:49.582227 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.582168 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhqrh"] Jul 08 17:34:49.583075 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.583046 2578 generic.go:356] "Generic (PLEG): container finished" podID="a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8" containerID="ab76796f2794ec44430a2128dcab4771befda4bd25c7c164721e1aba9c127232" exitCode=0 Jul 08 17:34:49.583194 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.583131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zpjm" event={"ID":"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8","Type":"ContainerDied","Data":"ab76796f2794ec44430a2128dcab4771befda4bd25c7c164721e1aba9c127232"} Jul 08 17:34:49.584393 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:49.584362 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461ece89_2553_4c64_8af2_80b73de11b77.slice/crio-463c77f3967d308f5c212e73686a6160289856930366de8771631ff4cf5dc04e WatchSource:0}: Error finding container 463c77f3967d308f5c212e73686a6160289856930366de8771631ff4cf5dc04e: Status 404 returned error can't find the container with id 463c77f3967d308f5c212e73686a6160289856930366de8771631ff4cf5dc04e Jul 08 17:34:49.584740 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.584716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4jl4" event={"ID":"71fef467-1e6d-45d5-b3fb-78ede7b5caff","Type":"ContainerStarted","Data":"c9070a604ccf7b038219fd7985d5da7c59acd6a51afcc605d69624dc9c31fecf"} Jul 08 17:34:49.585974 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:49.585948 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8877n" event={"ID":"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f","Type":"ContainerStarted","Data":"005a56651d3a8aea2a40bb2f0deb3f20e7a7e28cd0c47a49ce8ece8543598b56"} Jul 08 17:34:50.477465 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:50.477421 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:50.578035 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:50.577988 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:50.591369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:50.591335 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhqrh" event={"ID":"461ece89-2553-4c64-8af2-80b73de11b77","Type":"ContainerStarted","Data":"463c77f3967d308f5c212e73686a6160289856930366de8771631ff4cf5dc04e"} Jul 08 17:34:51.604553 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.604450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xxntr" event={"ID":"6cec3255-c25a-4c5e-ae32-65e5f5126624","Type":"ContainerStarted","Data":"42aae55feb20c5dcb5c360a903ec31e13b96f8b54a435778d058480e3aa0e836"} Jul 08 17:34:51.609896 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.609195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zpjm" event={"ID":"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8","Type":"ContainerStarted","Data":"7f78306c1ada91fc10b7d97bbb27a8697933d8f7f52e3c7e612304e8faf4e46f"} Jul 08 17:34:51.609896 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.609254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6zpjm" event={"ID":"a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8","Type":"ContainerStarted","Data":"94e483d1f8e5b940fc49f985c485791bdbc318c0205df15ab6bf4388cba2dd52"} Jul 08 17:34:51.623853 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.623418 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xxntr" podStartSLOduration=1.720910022 podStartE2EDuration="14.623399915s" totalImagesPullingTime="12.902489893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:34:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:51.622007718 +0000 UTC m=+67.919650882" watchObservedRunningTime="2026-07-08 17:34:51.623399915 +0000 UTC m=+67.921043086" Jul 08 17:34:51.641772 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.641718 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6zpjm" podStartSLOduration=3.307930233 podStartE2EDuration="4.64170232s" totalImagesPullingTime="1.250239288s" totalInitContainerRuntime="83.532799ms" isStatefulPod=true podCreationTimestamp="2026-07-08 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:51.640801472 +0000 UTC m=+67.938444639" watchObservedRunningTime="2026-07-08 17:34:51.64170232 +0000 UTC m=+67.939345486" Jul 08 17:34:51.798546 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.798347 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc"] Jul 08 17:34:51.801662 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.801639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:51.804926 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.804902 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7f5dd\"" Jul 08 17:34:51.805027 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.805007 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 17:34:51.811927 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.811871 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc"] Jul 08 17:34:51.834698 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.834603 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:51.906534 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.906491 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:51.932805 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.932763 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:51.932805 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:51.932809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58x6l\" (UniqueName: \"kubernetes.io/projected/49b64cca-249a-4df2-bc5e-cc33a2d57b86-kube-api-access-58x6l\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.033991 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.033956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.033991 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.033995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58x6l\" (UniqueName: \"kubernetes.io/projected/49b64cca-249a-4df2-bc5e-cc33a2d57b86-kube-api-access-58x6l\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.034235 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:52.034128 2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 08 17:34:52.034235 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:34:52.034200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert podName:49b64cca-249a-4df2-bc5e-cc33a2d57b86 nodeName:}" failed. No retries permitted until 2026-07-08 17:34:52.534179656 +0000 UTC m=+68.831822820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-wj6gc" (UID: "49b64cca-249a-4df2-bc5e-cc33a2d57b86") : secret "monitoring-plugin-cert" not found Jul 08 17:34:52.043883 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.043854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58x6l\" (UniqueName: \"kubernetes.io/projected/49b64cca-249a-4df2-bc5e-cc33a2d57b86-kube-api-access-58x6l\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.278702 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.278633 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:52.537276 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.537185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.540717 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.540684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/49b64cca-249a-4df2-bc5e-cc33a2d57b86-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-wj6gc\" (UID: \"49b64cca-249a-4df2-bc5e-cc33a2d57b86\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.556849 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.556816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gv4qs" Jul 08 17:34:52.620194 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.620071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4jl4" event={"ID":"71fef467-1e6d-45d5-b3fb-78ede7b5caff","Type":"ContainerStarted","Data":"f2446988ca4a9b3df6f755ea530ae43bfcab0be4e06967c2f50eb6c4f034cfb7"} Jul 08 17:34:52.620194 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.620142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g4jl4" event={"ID":"71fef467-1e6d-45d5-b3fb-78ede7b5caff","Type":"ContainerStarted","Data":"25ba8a361d5f495391ca32e294fd723037a51a1ab4944f1174e384fc1e094aa0"} Jul 08 17:34:52.645426 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.645230 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g4jl4" podStartSLOduration=66.451582529 podStartE2EDuration="1m8.645207816s" totalImagesPullingTime="2.193625287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:52.64349201 +0000 UTC m=+68.941135181" watchObservedRunningTime="2026-07-08 17:34:52.645207816 +0000 UTC m=+68.942850984" Jul 08 17:34:52.713015 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.712968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:52.785063 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.785013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:52.848890 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:52.848795 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:53.434620 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:53.434581 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:53.502805 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:53.502772 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:55.077429 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.077359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:55.168907 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.168871 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc"] Jul 08 17:34:55.172356 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:34:55.172331 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b64cca_249a_4df2_bc5e_cc33a2d57b86.slice/crio-2842b73345d5f204582bba7a5283cc0c671b923f6c3762d7c06ff5910e6ddb9c WatchSource:0}: Error finding container 2842b73345d5f204582bba7a5283cc0c671b923f6c3762d7c06ff5910e6ddb9c: Status 404 returned error can't find the container with id 2842b73345d5f204582bba7a5283cc0c671b923f6c3762d7c06ff5910e6ddb9c Jul 08 17:34:55.186773 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.186740 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:55.600445 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.600411 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:55.632550 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.632502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhqrh" event={"ID":"461ece89-2553-4c64-8af2-80b73de11b77","Type":"ContainerStarted","Data":"979904e6d215148ee8f4c900a91e774de4fbbd20efee0239d3f50dd194235b5d"} Jul 08 17:34:55.632726 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.632618 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:34:55.633688 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.633660 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" event={"ID":"49b64cca-249a-4df2-bc5e-cc33a2d57b86","Type":"ContainerStarted","Data":"2842b73345d5f204582bba7a5283cc0c671b923f6c3762d7c06ff5910e6ddb9c"} Jul 08 17:34:55.635030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.635009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8877n" event={"ID":"d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f","Type":"ContainerStarted","Data":"5e54cc6abc883ff452fd33b9b116da40b9c4918ea0a5f8918449663e49566035"} Jul 08 17:34:55.651810 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.651766 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zhqrh" podStartSLOduration=66.196485536 podStartE2EDuration="1m11.651750472s" totalImagesPullingTime="5.455264936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:55.651549658 +0000 UTC m=+71.949192827" watchObservedRunningTime="2026-07-08 17:34:55.651750472 +0000 UTC m=+71.949393639" Jul 08 17:34:55.667941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.667901 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8877n" podStartSLOduration=65.9792726 podStartE2EDuration="1m11.667888951s" totalImagesPullingTime="5.688616351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:55.66713475 +0000 UTC m=+71.964777916" watchObservedRunningTime="2026-07-08 17:34:55.667888951 +0000 UTC m=+71.965532115" Jul 08 17:34:55.672222 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.672202 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:55.733958 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:55.733930 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:57.645975 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:57.645939 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" event={"ID":"49b64cca-249a-4df2-bc5e-cc33a2d57b86","Type":"ContainerStarted","Data":"72ae096a4e5ea48e8c8d70cd35e517dcbbc38d007fa1ce081a8af653c1e3e8cf"} Jul 08 17:34:57.646399 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:57.646157 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:57.650825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:57.650804 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" Jul 08 17:34:57.663632 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:57.663585 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-wj6gc" podStartSLOduration=4.829117003 podStartE2EDuration="6.663570025s" totalImagesPullingTime="1.834453022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:34:57.662779844 +0000 UTC m=+73.960423012" watchObservedRunningTime="2026-07-08 17:34:57.663570025 +0000 UTC m=+73.961213191" Jul 08 17:34:58.542908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:58.542872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:58.687637 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:58.687599 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:34:58.787507 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:34:58.787479 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:02.440851 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:02.440815 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:02.495988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:02.495955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:10.779116 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:10.778986 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:10.845186 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:10.845151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:15.210241 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:15.210206 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:20.603033 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:20.602980 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:20.668203 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:20.668164 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:25.075022 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:25.074977 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:25.176776 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:25.176742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:26.641342 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:26.641304 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zhqrh" Jul 08 17:35:36.323999 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:36.323956 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-679nq_b02a4077-3343-4838-b39a-04e554934119/serve-healthcheck-canary/0.log" Jul 08 17:35:36.491616 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:36.491582 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:43.060100 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:43.060056 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:35:43.143904 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:35:43.143869 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:06.782383 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:06.782335 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:22.140713 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:22.140677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:22.198918 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:22.198894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:27.677106 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:27.677071 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:27.745264 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:27.745235 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:30.110947 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:30.110901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:30.209010 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:30.208974 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:30.270773 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:30.270733 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:30.480168 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:30.480085 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:30.866471 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:30.866445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:31.686355 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:31.686324 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:31.792199 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:31.792168 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:32.497059 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.497029 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:36:32.498800 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.498784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.502251 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502230 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qq47s\"" Jul 08 17:36:32.502378 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 17:36:32.502378 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502364 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 17:36:32.502495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502382 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 17:36:32.502495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502384 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 17:36:32.502495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502312 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 17:36:32.502495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502493 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 17:36:32.502780 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.502235 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 17:36:32.509763 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.509743 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:36:32.605164 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkmc\" (UniqueName: \"kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.605298 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.605298 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.605298 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.605394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.605394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.605326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706451 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706451 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706983 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkmc\" (UniqueName: \"kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706983 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706983 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.706983 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.706583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.707216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.707140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.707216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.707181 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.707329 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.707300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.709084 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.709062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.709179 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.709062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.716753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.716733 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkmc\" (UniqueName: \"kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc\") pod \"console-68f8d974f6-v4qcx\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.807846 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.807751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:32.895480 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.895447 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:32.938679 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.938656 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:36:32.941190 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:36:32.941166 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc454749f_bf1b_4551_8e8f_3159ee3b5d00.slice/crio-259f8ba437304c7259651fc53ac67dea9be1c78dfccbaf4305fd9a2f5180505d WatchSource:0}: Error finding container 259f8ba437304c7259651fc53ac67dea9be1c78dfccbaf4305fd9a2f5180505d: Status 404 returned error can't find the container with id 259f8ba437304c7259651fc53ac67dea9be1c78dfccbaf4305fd9a2f5180505d Jul 08 17:36:32.956025 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.955998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f8d974f6-v4qcx" event={"ID":"c454749f-bf1b-4551-8e8f-3159ee3b5d00","Type":"ContainerStarted","Data":"259f8ba437304c7259651fc53ac67dea9be1c78dfccbaf4305fd9a2f5180505d"} Jul 08 17:36:32.998311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:32.998285 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:33.755872 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:33.755836 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:33.856627 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:33.856547 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:33.957191 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:33.957153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:34.684974 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.684937 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:34.785415 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.785374 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:34.890030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.889978 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:36:34.892276 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.892250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.900913 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.900730 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 17:36:34.903973 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.903952 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:36:34.923239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.923409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.923409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.923409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.923409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.924022 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdk9m\" (UniqueName: \"kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:34.924022 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:34.923551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024540 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024540 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024540 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdk9m\" (UniqueName: \"kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.024825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.024672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.025252 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.025219 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.025955 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.025931 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.026072 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.026058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.026462 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.026412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.027922 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.027903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.028073 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.028050 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.033543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.033497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdk9m\" (UniqueName: \"kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m\") pod \"console-774ccb7c86-8kt4d\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.205035 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.204995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:35.536052 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.536018 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:36:35.539027 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:36:35.539002 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403f6f2f_dbfd_4e68_8c63_a9b14e1f4554.slice/crio-a46de20dbe0f5e9a0875dab9aec645c6e7ebff58074690e8dd0e9359313572a3 WatchSource:0}: Error finding container a46de20dbe0f5e9a0875dab9aec645c6e7ebff58074690e8dd0e9359313572a3: Status 404 returned error can't find the container with id a46de20dbe0f5e9a0875dab9aec645c6e7ebff58074690e8dd0e9359313572a3 Jul 08 17:36:35.967447 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.967409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774ccb7c86-8kt4d" event={"ID":"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554","Type":"ContainerStarted","Data":"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc"} Jul 08 17:36:35.967938 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.967470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774ccb7c86-8kt4d" event={"ID":"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554","Type":"ContainerStarted","Data":"a46de20dbe0f5e9a0875dab9aec645c6e7ebff58074690e8dd0e9359313572a3"} Jul 08 17:36:35.968931 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.968907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f8d974f6-v4qcx" event={"ID":"c454749f-bf1b-4551-8e8f-3159ee3b5d00","Type":"ContainerStarted","Data":"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920"} Jul 08 17:36:35.986226 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:35.986180 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774ccb7c86-8kt4d" podStartSLOduration=1.986167215 podStartE2EDuration="1.986167215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:36:35.985179395 +0000 UTC m=+172.282822562" watchObservedRunningTime="2026-07-08 17:36:35.986167215 +0000 UTC m=+172.283810382" Jul 08 17:36:36.003365 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:36.003319 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f8d974f6-v4qcx" podStartSLOduration=1.511315304 podStartE2EDuration="4.003305526s" totalImagesPullingTime="2.491990222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:36:36.002070972 +0000 UTC m=+172.299714139" watchObservedRunningTime="2026-07-08 17:36:36.003305526 +0000 UTC m=+172.300948693" Jul 08 17:36:36.769558 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:36.769499 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:36.868582 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:36.868544 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:39.696998 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:39.696964 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:39.758043 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:39.758008 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:40.086609 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:40.086501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:40.156109 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:40.156067 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:40.247950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:40.247923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:40.310796 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:40.310757 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:42.807951 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:42.807910 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:42.808366 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:42.807980 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:42.812553 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:42.812532 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:42.995134 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:42.995096 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:36:43.128075 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:43.128045 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:43.229693 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:43.229661 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:44.039994 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:44.039940 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:45.205990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:45.205950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:45.206361 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:45.206022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:45.210740 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:45.210715 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:46.003870 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:46.003844 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:36:46.049313 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:46.049273 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:36:46.148768 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:46.148734 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:46.202428 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:46.202386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:46.264038 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:46.263959 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:55.069536 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:55.069486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:36:55.170663 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:36:55.170634 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:37:06.444503 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:06.444468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:37:06.569503 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:06.569468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:37:11.069457 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.069415 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-68f8d974f6-v4qcx" podUID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" containerName="console" containerID="cri-o://166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920" gracePeriod=15 Jul 08 17:37:11.296719 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.296697 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:37:11.383050 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383019 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383057 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383137 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383169 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383203 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lkmc\" (UniqueName: \"kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc\") pod \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\" (UID: \"c454749f-bf1b-4551-8e8f-3159ee3b5d00\") " Jul 08 17:37:11.383578 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383551 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:37:11.383650 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383616 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca" (OuterVolumeSpecName: "service-ca") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:37:11.383705 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.383672 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config" (OuterVolumeSpecName: "console-config") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:37:11.385557 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.385508 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc" (OuterVolumeSpecName: "kube-api-access-7lkmc") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "kube-api-access-7lkmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:37:11.385557 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.385546 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:37:11.385689 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.385607 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c454749f-bf1b-4551-8e8f-3159ee3b5d00" (UID: "c454749f-bf1b-4551-8e8f-3159ee3b5d00"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:37:11.483798 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483757 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:11.483798 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483792 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:11.483798 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483803 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:11.483798 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483812 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:11.484030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483819 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c454749f-bf1b-4551-8e8f-3159ee3b5d00-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:11.484030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:11.483828 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lkmc\" (UniqueName: \"kubernetes.io/projected/c454749f-bf1b-4551-8e8f-3159ee3b5d00-kube-api-access-7lkmc\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:37:12.085162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.085130 2578 generic.go:356] "Generic (PLEG): container finished" podID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" containerID="166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920" exitCode=0 Jul 08 17:37:12.085560 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.085181 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f8d974f6-v4qcx" Jul 08 17:37:12.085560 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.085221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f8d974f6-v4qcx" event={"ID":"c454749f-bf1b-4551-8e8f-3159ee3b5d00","Type":"ContainerDied","Data":"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920"} Jul 08 17:37:12.085560 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.085277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f8d974f6-v4qcx" event={"ID":"c454749f-bf1b-4551-8e8f-3159ee3b5d00","Type":"ContainerDied","Data":"259f8ba437304c7259651fc53ac67dea9be1c78dfccbaf4305fd9a2f5180505d"} Jul 08 17:37:12.085560 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.085316 2578 scope.go:117] "RemoveContainer" containerID="166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920" Jul 08 17:37:12.095212 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.094933 2578 scope.go:117] "RemoveContainer" containerID="166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920" Jul 08 17:37:12.095481 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:37:12.095456 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920\": container with ID starting with 166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920 not found: ID does not exist" containerID="166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920" Jul 08 17:37:12.095553 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.095490 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920"} err="failed to get container status \"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920\": rpc error: code = NotFound desc = could not find container \"166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920\": container with ID starting with 166ce06f56869151079faea4f470d9928da0025a2749631e4ae1361421817920 not found: ID does not exist" Jul 08 17:37:12.108775 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.108756 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:37:12.111000 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.110976 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68f8d974f6-v4qcx"] Jul 08 17:37:12.309059 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:12.309016 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" path="/var/lib/kubelet/pods/c454749f-bf1b-4551-8e8f-3159ee3b5d00/volumes" Jul 08 17:37:53.097120 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:37:53.097070 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:38:44.105716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:38:44.105684 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 17:39:50.016831 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:50.016791 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:51.238109 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.238078 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 17:39:51.238494 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.238303 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" containerName="console" Jul 08 17:39:51.238494 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.238314 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" containerName="console" Jul 08 17:39:51.238494 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.238369 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c454749f-bf1b-4551-8e8f-3159ee3b5d00" containerName="console" Jul 08 17:39:51.242451 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.242430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.244588 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244566 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 17:39:51.244855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244835 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 17:39:51.244933 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244855 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 17:39:51.244933 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244888 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-799rc\"" Jul 08 17:39:51.244933 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 17:39:51.244933 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244913 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 17:39:51.245118 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.244943 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 17:39:51.245118 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.245111 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 17:39:51.245194 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.245139 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 17:39:51.251242 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.251222 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 17:39:51.267706 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.267686 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 17:39:51.386024 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.385983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386170 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386170 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386170 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-kube-api-access-gbt4c\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386170 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386170 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-web-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-out\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386409 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386608 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.386608 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.386457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.447387 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.447362 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:51.486818 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.486911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.486911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.486911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.486911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:39:51.486971 2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.486987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-kube-api-access-gbt4c\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:39:51.487043 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls podName:2f1f9ea7-d706-450f-818b-49f3668be1c7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:51.987026083 +0000 UTC m=+368.284669232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "2f1f9ea7-d706-450f-818b-49f3668be1c7") : secret "alertmanager-main-tls" not found Jul 08 17:39:51.487103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-web-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487465 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-out\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487465 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487465 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487679 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487739 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487683 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.487952 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.487833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f1f9ea7-d706-450f-818b-49f3668be1c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490365 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490530 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-out\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490713 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490759 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.490910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.491022 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.490965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-web-config\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.491227 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.491205 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.495811 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.495794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbt4c\" (UniqueName: \"kubernetes.io/projected/2f1f9ea7-d706-450f-818b-49f3668be1c7-kube-api-access-gbt4c\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.624805 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.624768 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:51.989692 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.989664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:51.992041 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:51.992014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2f1f9ea7-d706-450f-818b-49f3668be1c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2f1f9ea7-d706-450f-818b-49f3668be1c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:52.152436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:52.152407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 17:39:52.293478 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:52.293454 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 17:39:52.295007 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:39:52.294984 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1f9ea7_d706_450f_818b_49f3668be1c7.slice/crio-70bdb3921472c201543c7af8f02a82c2464b488e126c78945c84da7fcfd10c90 WatchSource:0}: Error finding container 70bdb3921472c201543c7af8f02a82c2464b488e126c78945c84da7fcfd10c90: Status 404 returned error can't find the container with id 70bdb3921472c201543c7af8f02a82c2464b488e126c78945c84da7fcfd10c90 Jul 08 17:39:52.296927 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:52.296907 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:39:52.593719 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:52.593635 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"70bdb3921472c201543c7af8f02a82c2464b488e126c78945c84da7fcfd10c90"} Jul 08 17:39:53.112348 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:53.112316 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.213328 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:53.213294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.475797 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:53.475714 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.618334 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:53.618294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.600745 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:54.600710 2578 generic.go:356] "Generic (PLEG): container finished" podID="2f1f9ea7-d706-450f-818b-49f3668be1c7" containerID="3f7a4cdb78b4c7b5dae2b4aac7c6753af6dca503d7af9a80ab0406f4aaa7038f" exitCode=0 Jul 08 17:39:54.601239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:54.600778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerDied","Data":"3f7a4cdb78b4c7b5dae2b4aac7c6753af6dca503d7af9a80ab0406f4aaa7038f"} Jul 08 17:39:56.387938 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.387894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.389237 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.389210 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 17:39:56.393715 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.393685 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.397141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.396787 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 17:39:56.397141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.396805 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 17:39:56.397141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.396812 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 17:39:56.397141 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397086 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397750 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f26rm\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397773 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397773 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397839 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397781 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397851 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.397947 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.398014 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.398027 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-74gaimbbdf0k8\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.398036 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 17:39:56.398153 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.398071 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 17:39:56.400907 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.400715 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 17:39:56.403564 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.403505 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 17:39:56.413454 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.413429 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 17:39:56.422259 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422376 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qgh\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-kube-api-access-p9qgh\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422376 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422376 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422672 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422672 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422672 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422672 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422631 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.422863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.423106 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.423106 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.422919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523532 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qgh\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-kube-api-access-p9qgh\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.523720 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.523985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524053 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524127 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.524762 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.524681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.525417 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.525062 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.525999 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.525938 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.526173 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.526149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.526825 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.526794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.527953 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.527883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.528483 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.528172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d1b9d204-def9-4356-80f4-cbd8a0606f99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.528483 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.528334 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.528483 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.528353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1b9d204-def9-4356-80f4-cbd8a0606f99-config-out\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.528483 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.528428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.529077 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.529051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.529990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.529553 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.530448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.530405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-web-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.530999 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.530976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.531123 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.531071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-config\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.532810 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.532787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.533067 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.533036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d1b9d204-def9-4356-80f4-cbd8a0606f99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.533140 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.533075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qgh\" (UniqueName: \"kubernetes.io/projected/d1b9d204-def9-4356-80f4-cbd8a0606f99-kube-api-access-p9qgh\") pod \"prometheus-k8s-0\" (UID: \"d1b9d204-def9-4356-80f4-cbd8a0606f99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.706883 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.706796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:39:56.860139 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:56.860116 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 17:39:56.862137 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:39:56.862111 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b9d204_def9_4356_80f4_cbd8a0606f99.slice/crio-fb78d81089736759c95f990c8a8ac9b60192b2aad7b826a1fe7e0830a76cdfb1 WatchSource:0}: Error finding container fb78d81089736759c95f990c8a8ac9b60192b2aad7b826a1fe7e0830a76cdfb1: Status 404 returned error can't find the container with id fb78d81089736759c95f990c8a8ac9b60192b2aad7b826a1fe7e0830a76cdfb1 Jul 08 17:39:57.612797 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.612769 2578 generic.go:356] "Generic (PLEG): container finished" podID="d1b9d204-def9-4356-80f4-cbd8a0606f99" containerID="2421b7ad377f6fc6f65fab5961b1ec51a6a0da0c08940935452df16723812078" exitCode=0 Jul 08 17:39:57.613179 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.612846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerDied","Data":"2421b7ad377f6fc6f65fab5961b1ec51a6a0da0c08940935452df16723812078"} Jul 08 17:39:57.613179 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.612916 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"fb78d81089736759c95f990c8a8ac9b60192b2aad7b826a1fe7e0830a76cdfb1"} Jul 08 17:39:57.615390 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.615370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"27ecaee0281b9453f337fe800978cf5fe9627fe8690c7113e8138e514b90b2ef"} Jul 08 17:39:57.615467 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.615421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"5819dd0844a7e1aa865a09686079391ad19a81f45dad6355e20008b7f0d01477"} Jul 08 17:39:57.615467 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.615446 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"9688975f386a75a503740dc8a8ded524c75632f5ce4905695afd717264dce2a4"} Jul 08 17:39:57.615548 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:57.615470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"a2c67aed7d3c9bb22266b266dddf41e12e2a688f31056b2efc5dc29ac9ed7466"} Jul 08 17:39:58.624137 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:58.624038 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"b9256a105548fec692c23916d1556b846d156d255068a5b11106023159fe3c2e"} Jul 08 17:39:58.624137 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:58.624130 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2f1f9ea7-d706-450f-818b-49f3668be1c7","Type":"ContainerStarted","Data":"295c2a41db2b61c35fa2c4700b5f60cc51b7b85674085e6e56be2b3e40fdc678"} Jul 08 17:39:58.653762 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:39:58.653405 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.48099189 podStartE2EDuration="7.653385672s" totalImagesPullingTime="5.156142254s" totalInitContainerRuntime="16.251528ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:58.6511198 +0000 UTC m=+374.948762979" watchObservedRunningTime="2026-07-08 17:39:58.653385672 +0000 UTC m=+374.951028840" Jul 08 17:40:01.154060 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.154017 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:40:01.157426 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.157405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.168468 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.168440 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:40:01.212644 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.212616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:01.265438 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265583 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265462 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265583 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265671 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265671 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265671 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chl9s\" (UniqueName: \"kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.265788 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.265691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366493 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chl9s\" (UniqueName: \"kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366873 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.366873 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.366716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.367372 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.367319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.367496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.367419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.367651 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.367498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.367941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.367916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.369900 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.369881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.370115 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.370093 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.377086 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.377038 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chl9s\" (UniqueName: \"kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s\") pod \"console-6b9599464c-vwl2r\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.467592 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.467464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:01.639045 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.639011 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:40:01.639235 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.639104 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"19cf6507aa71b670d859a9e4ee7f839e16f8ca5c2eb587c4d794c1b9b4899953"} Jul 08 17:40:01.639235 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:01.639168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"0a801766cd707b275df86f16f02b256740306bfc14198c5dc76a1aaa9e7928fa"} Jul 08 17:40:01.642103 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:40:01.642075 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f76537c_104a_47e1_9cb2_917f5a00c2af.slice/crio-1a3015912aa414a15c5c7938775183f8ebf1e0bff66206b98adb4c2a43aa78c9 WatchSource:0}: Error finding container 1a3015912aa414a15c5c7938775183f8ebf1e0bff66206b98adb4c2a43aa78c9: Status 404 returned error can't find the container with id 1a3015912aa414a15c5c7938775183f8ebf1e0bff66206b98adb4c2a43aa78c9 Jul 08 17:40:02.645534 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:02.645432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9599464c-vwl2r" event={"ID":"7f76537c-104a-47e1-9cb2-917f5a00c2af","Type":"ContainerStarted","Data":"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4"} Jul 08 17:40:02.645534 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:02.645496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9599464c-vwl2r" event={"ID":"7f76537c-104a-47e1-9cb2-917f5a00c2af","Type":"ContainerStarted","Data":"1a3015912aa414a15c5c7938775183f8ebf1e0bff66206b98adb4c2a43aa78c9"} Jul 08 17:40:02.663581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:02.663505 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b9599464c-vwl2r" podStartSLOduration=1.6634769390000002 podStartE2EDuration="1.663476939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:02.662104016 +0000 UTC m=+378.959747185" watchObservedRunningTime="2026-07-08 17:40:02.663476939 +0000 UTC m=+378.961120107" Jul 08 17:40:04.657726 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:04.657688 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"d9d643d89c860bbbbf08da9f8a4993781426183802558f36c5827df39f6c2f20"} Jul 08 17:40:04.658121 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:04.657765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"68acc45427a91b10ebd85dd09923db19cf8f6d62d2261f95fcd19660efa124e5"} Jul 08 17:40:04.658121 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:04.657806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"9aa4a21ba912eab1545dc16493f4e2e29bdb90c7281076f8ec4f6c924376a127"} Jul 08 17:40:04.658121 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:04.657849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d1b9d204-def9-4356-80f4-cbd8a0606f99","Type":"ContainerStarted","Data":"58560498dbb0683732e3f76ecac2841a9e7bee191bd77e5746b48d40f002714d"} Jul 08 17:40:04.688547 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:04.688476 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.685897139 podStartE2EDuration="8.688454425s" totalImagesPullingTime="5.980053954s" totalInitContainerRuntime="22.503332ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:04.687121665 +0000 UTC m=+380.984764831" watchObservedRunningTime="2026-07-08 17:40:04.688454425 +0000 UTC m=+380.986097592" Jul 08 17:40:06.707411 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:06.707376 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:40:11.468552 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.468501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:11.468916 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.468577 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:11.473295 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.473274 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:11.686797 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.686769 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:40:11.733867 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.733783 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:40:11.852526 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.852446 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:11.953452 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:11.953424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:12.009178 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:12.009101 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:36.759154 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:36.759087 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-774ccb7c86-8kt4d" podUID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" containerName="console" containerID="cri-o://e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc" gracePeriod=15 Jul 08 17:40:37.001349 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.001325 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:40:37.072171 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072171 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072171 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072139 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072435 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072273 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072435 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072333 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072435 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072435 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072405 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdk9m\" (UniqueName: \"kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m\") pod \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\" (UID: \"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554\") " Jul 08 17:40:37.072673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072550 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca" (OuterVolumeSpecName: "service-ca") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:37.072673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072561 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:37.072673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072600 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config" (OuterVolumeSpecName: "console-config") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:37.072802 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072776 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:37.072836 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072807 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.072836 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072830 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.072909 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.072847 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-trusted-ca-bundle\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.074688 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.074646 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m" (OuterVolumeSpecName: "kube-api-access-sdk9m") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "kube-api-access-sdk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:40:37.074792 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.074705 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:40:37.074792 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.074733 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" (UID: "403f6f2f-dbfd-4e68-8c63-a9b14e1f4554"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:40:37.173684 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.173650 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.173684 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.173678 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdk9m\" (UniqueName: \"kubernetes.io/projected/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-kube-api-access-sdk9m\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.173684 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.173687 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.173684 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.173696 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:40:37.779674 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.779639 2578 generic.go:356] "Generic (PLEG): container finished" podID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" containerID="e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc" exitCode=0 Jul 08 17:40:37.780136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.779681 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774ccb7c86-8kt4d" event={"ID":"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554","Type":"ContainerDied","Data":"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc"} Jul 08 17:40:37.780136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.779684 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774ccb7c86-8kt4d" Jul 08 17:40:37.780136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.779724 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774ccb7c86-8kt4d" event={"ID":"403f6f2f-dbfd-4e68-8c63-a9b14e1f4554","Type":"ContainerDied","Data":"a46de20dbe0f5e9a0875dab9aec645c6e7ebff58074690e8dd0e9359313572a3"} Jul 08 17:40:37.780136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.779748 2578 scope.go:117] "RemoveContainer" containerID="e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc" Jul 08 17:40:37.788459 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.788441 2578 scope.go:117] "RemoveContainer" containerID="e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc" Jul 08 17:40:37.788726 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:40:37.788705 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc\": container with ID starting with e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc not found: ID does not exist" containerID="e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc" Jul 08 17:40:37.788832 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.788733 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc"} err="failed to get container status \"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc\": rpc error: code = NotFound desc = could not find container \"e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc\": container with ID starting with e6563af0ba1a6d0720954b5dafb5956d1bfa0e0508dde2200b1dde6e4cce14cc not found: ID does not exist" Jul 08 17:40:37.801567 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.801545 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:40:37.805391 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:37.805370 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774ccb7c86-8kt4d"] Jul 08 17:40:38.308417 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:38.308386 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" path="/var/lib/kubelet/pods/403f6f2f-dbfd-4e68-8c63-a9b14e1f4554/volumes" Jul 08 17:40:56.707587 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:56.707549 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:40:56.727757 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:56.727729 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:40:56.867285 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:40:56.867258 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 17:41:06.308233 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:06.308154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:06.409725 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:06.409698 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:18.279768 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:18.279734 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:18.336615 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:18.336588 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:22.415003 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:22.414971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:25.076322 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:25.076288 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:25.176620 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:25.176582 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:25.409470 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:25.409445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:30.135459 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.135425 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:41:30.135871 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.135722 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" containerName="console" Jul 08 17:41:30.135871 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.135734 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" containerName="console" Jul 08 17:41:30.135871 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.135790 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="403f6f2f-dbfd-4e68-8c63-a9b14e1f4554" containerName="console" Jul 08 17:41:30.138620 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.138605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.154120 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.154097 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:41:30.186279 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.186251 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:30.265155 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265335 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265335 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265335 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpcp\" (UniqueName: \"kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265335 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265335 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.265577 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.265388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366651 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366651 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpcp\" (UniqueName: \"kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.366897 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.366862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.367504 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.367460 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.367871 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.367500 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.367871 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.367550 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.368001 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.367643 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.369420 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.369398 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.369473 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.369407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.375059 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.375041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpcp\" (UniqueName: \"kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp\") pod \"console-5fb4d5787d-g4cvq\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.447741 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.447643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:30.576879 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.576847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:41:30.579931 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:41:30.579899 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38702120_936a_4f53_ad69_3b30289d5eaa.slice/crio-6faf6398dce3186c02f82570ce92f03d6ed188ad76a9cb003f8f6a271db17ffa WatchSource:0}: Error finding container 6faf6398dce3186c02f82570ce92f03d6ed188ad76a9cb003f8f6a271db17ffa: Status 404 returned error can't find the container with id 6faf6398dce3186c02f82570ce92f03d6ed188ad76a9cb003f8f6a271db17ffa Jul 08 17:41:30.969683 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.969651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4d5787d-g4cvq" event={"ID":"38702120-936a-4f53-ad69-3b30289d5eaa","Type":"ContainerStarted","Data":"3b6224ba5e388fba7236b4dfbc2a5daf185a963578223bada57d1edc1cdbaabf"} Jul 08 17:41:30.969884 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.969708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4d5787d-g4cvq" event={"ID":"38702120-936a-4f53-ad69-3b30289d5eaa","Type":"ContainerStarted","Data":"6faf6398dce3186c02f82570ce92f03d6ed188ad76a9cb003f8f6a271db17ffa"} Jul 08 17:41:30.988665 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:30.988616 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fb4d5787d-g4cvq" podStartSLOduration=0.988599599 podStartE2EDuration="988.599599ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:41:30.988316586 +0000 UTC m=+467.285959754" watchObservedRunningTime="2026-07-08 17:41:30.988599599 +0000 UTC m=+467.286242767" Jul 08 17:41:40.447786 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:40.447732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:40.448295 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:40.447819 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:40.453198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:40.453174 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:41.009312 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:41.009283 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:41:41.056717 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:41.056680 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:41:41.177206 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:41.177176 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:41.279133 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:41.279050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:41.378917 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:41:41.378884 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:04.352132 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:04.352096 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:04.435230 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:04.435199 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:06.082310 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.082270 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6b9599464c-vwl2r" podUID="7f76537c-104a-47e1-9cb2-917f5a00c2af" containerName="console" containerID="cri-o://987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4" gracePeriod=15 Jul 08 17:42:06.321637 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.321610 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:42:06.354243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354249 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354282 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354316 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354479 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354399 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chl9s\" (UniqueName: \"kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354479 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354435 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354479 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354456 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert\") pod \"7f76537c-104a-47e1-9cb2-917f5a00c2af\" (UID: \"7f76537c-104a-47e1-9cb2-917f5a00c2af\") " Jul 08 17:42:06.354646 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354624 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:42:06.354696 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354686 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.354753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354713 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:42:06.354753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.354737 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:42:06.355079 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.355037 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config" (OuterVolumeSpecName: "console-config") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:42:06.356806 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.356784 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s" (OuterVolumeSpecName: "kube-api-access-chl9s") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "kube-api-access-chl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:42:06.356806 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.356791 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:42:06.356944 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.356800 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f76537c-104a-47e1-9cb2-917f5a00c2af" (UID: "7f76537c-104a-47e1-9cb2-917f5a00c2af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.455828 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.455878 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chl9s\" (UniqueName: \"kubernetes.io/projected/7f76537c-104a-47e1-9cb2-917f5a00c2af-kube-api-access-chl9s\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.455895 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.455910 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f76537c-104a-47e1-9cb2-917f5a00c2af-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.455966 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-trusted-ca-bundle\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:06.459911 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:06.456000 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f76537c-104a-47e1-9cb2-917f5a00c2af-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:42:07.096377 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.096341 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f76537c-104a-47e1-9cb2-917f5a00c2af" containerID="987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4" exitCode=0 Jul 08 17:42:07.096801 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.096403 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b9599464c-vwl2r" Jul 08 17:42:07.096801 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.096422 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9599464c-vwl2r" event={"ID":"7f76537c-104a-47e1-9cb2-917f5a00c2af","Type":"ContainerDied","Data":"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4"} Jul 08 17:42:07.096801 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.096495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b9599464c-vwl2r" event={"ID":"7f76537c-104a-47e1-9cb2-917f5a00c2af","Type":"ContainerDied","Data":"1a3015912aa414a15c5c7938775183f8ebf1e0bff66206b98adb4c2a43aa78c9"} Jul 08 17:42:07.096801 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.096538 2578 scope.go:117] "RemoveContainer" containerID="987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4" Jul 08 17:42:07.105412 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.105394 2578 scope.go:117] "RemoveContainer" containerID="987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4" Jul 08 17:42:07.105673 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:42:07.105656 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4\": container with ID starting with 987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4 not found: ID does not exist" containerID="987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4" Jul 08 17:42:07.105733 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.105681 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4"} err="failed to get container status \"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4\": rpc error: code = NotFound desc = could not find container \"987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4\": container with ID starting with 987cc1a63ad4f3f31f1735eb39bb5e5a81bc23507ecf2ca1cedb39a5b839c1a4 not found: ID does not exist" Jul 08 17:42:07.121179 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.121159 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:42:07.124600 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:07.124580 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b9599464c-vwl2r"] Jul 08 17:42:08.308567 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:08.308506 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f76537c-104a-47e1-9cb2-917f5a00c2af" path="/var/lib/kubelet/pods/7f76537c-104a-47e1-9cb2-917f5a00c2af/volumes" Jul 08 17:42:31.359042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:31.359012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:32.720084 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:32.720048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:35.060258 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:35.060176 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:35.148227 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:35.148194 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:36.501656 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:36.501622 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:36.610108 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:36.610074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:38.864826 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:38.864799 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:39.094347 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:39.094308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:41.774321 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:41.774286 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:41.833172 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:41.833143 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:46.071020 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:46.070987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:49.453248 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:49.453214 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:49.570284 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:49.570250 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:51.281841 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.281807 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs"] Jul 08 17:42:51.282211 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.282091 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f76537c-104a-47e1-9cb2-917f5a00c2af" containerName="console" Jul 08 17:42:51.282211 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.282102 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f76537c-104a-47e1-9cb2-917f5a00c2af" containerName="console" Jul 08 17:42:51.282211 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.282171 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f76537c-104a-47e1-9cb2-917f5a00c2af" containerName="console" Jul 08 17:42:51.286144 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.286127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.288547 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.288528 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 17:42:51.288635 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.288604 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:42:51.288710 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.288693 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-b9rzj\"" Jul 08 17:42:51.294619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.294585 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs"] Jul 08 17:42:51.407308 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.407269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/521be8a3-2681-473c-9140-cbadba624596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.407493 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.407369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfgb\" (UniqueName: \"kubernetes.io/projected/521be8a3-2681-473c-9140-cbadba624596-kube-api-access-vlfgb\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.507855 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.507820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/521be8a3-2681-473c-9140-cbadba624596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.508021 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.507885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfgb\" (UniqueName: \"kubernetes.io/projected/521be8a3-2681-473c-9140-cbadba624596-kube-api-access-vlfgb\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.508207 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.508187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/521be8a3-2681-473c-9140-cbadba624596-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.516300 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.516273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfgb\" (UniqueName: \"kubernetes.io/projected/521be8a3-2681-473c-9140-cbadba624596-kube-api-access-vlfgb\") pod \"openshift-lws-operator-fd8ccff4c-xjpzs\" (UID: \"521be8a3-2681-473c-9140-cbadba624596\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.608485 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.608452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" Jul 08 17:42:51.739203 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:51.739172 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs"] Jul 08 17:42:51.742182 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:42:51.742156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521be8a3_2681_473c_9140_cbadba624596.slice/crio-849035aee19db3256e1963ce62f96fc65f272d454ea35f14533863ac59ff60df WatchSource:0}: Error finding container 849035aee19db3256e1963ce62f96fc65f272d454ea35f14533863ac59ff60df: Status 404 returned error can't find the container with id 849035aee19db3256e1963ce62f96fc65f272d454ea35f14533863ac59ff60df Jul 08 17:42:52.257294 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:52.257262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" event={"ID":"521be8a3-2681-473c-9140-cbadba624596","Type":"ContainerStarted","Data":"849035aee19db3256e1963ce62f96fc65f272d454ea35f14533863ac59ff60df"} Jul 08 17:42:54.260966 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:54.260912 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:54.323640 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:54.323603 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:55.271289 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:55.271256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" event={"ID":"521be8a3-2681-473c-9140-cbadba624596","Type":"ContainerStarted","Data":"2a095db12f60f5f454258deb6cb37d3d451e0a40c0952c9c3893b8cea8e57ca5"} Jul 08 17:42:55.287487 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:55.287441 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjpzs" podStartSLOduration=1.376080395 podStartE2EDuration="4.287425951s" totalImagesPullingTime="2.911345556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:42:55.286751908 +0000 UTC m=+551.584395076" watchObservedRunningTime="2026-07-08 17:42:55.287425951 +0000 UTC m=+551.585069117" Jul 08 17:42:55.959778 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:55.959730 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:56.060559 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:56.060506 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:57.680157 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:42:57.680092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:05.864103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:05.864064 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:11.607215 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:11.607175 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:11.714204 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:11.714171 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.036050 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.035956 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.213229 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.213190 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.315017 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.314931 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.382378 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.382336 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5"] Jul 08 17:43:12.385475 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.385453 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.388796 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.388776 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 17:43:12.388920 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.388904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cfbk9\"" Jul 08 17:43:12.389253 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.389232 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 17:43:12.389327 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.389267 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 17:43:12.389502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.389490 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 17:43:12.408267 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.408235 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5"] Jul 08 17:43:12.479700 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.479664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.479700 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.479705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.479910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.479769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.479910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.479827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.479910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.479851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpl8\" (UniqueName: \"kubernetes.io/projected/86e394c9-4229-4cf1-b75f-98672d79b8d1-kube-api-access-dgpl8\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.522134 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.522111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.581218 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.581218 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpl8\" (UniqueName: \"kubernetes.io/projected/86e394c9-4229-4cf1-b75f-98672d79b8d1-kube-api-access-dgpl8\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.581218 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.581428 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.581428 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.581615 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.581595 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-manifests\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.583703 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.583676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86e394c9-4229-4cf1-b75f-98672d79b8d1-tmp\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.583802 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.583754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.583854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.583840 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86e394c9-4229-4cf1-b75f-98672d79b8d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.590959 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.590924 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpl8\" (UniqueName: \"kubernetes.io/projected/86e394c9-4229-4cf1-b75f-98672d79b8d1-kube-api-access-dgpl8\") pod \"opendatahub-operator-controller-manager-5c548d5447-7plc5\" (UID: \"86e394c9-4229-4cf1-b75f-98672d79b8d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.608221 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.608197 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.695700 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.695666 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:12.715295 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.715267 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.801540 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.801499 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.830135 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.830008 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5"] Jul 08 17:43:12.833471 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:43:12.833401 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e394c9_4229_4cf1_b75f_98672d79b8d1.slice/crio-91d741d8be768c747da0ff658ca5bb55ad6155981e8278eb61a69daf894abc2d WatchSource:0}: Error finding container 91d741d8be768c747da0ff658ca5bb55ad6155981e8278eb61a69daf894abc2d: Status 404 returned error can't find the container with id 91d741d8be768c747da0ff658ca5bb55ad6155981e8278eb61a69daf894abc2d Jul 08 17:43:12.864307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.864264 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:12.964386 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:12.964357 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:13.336122 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:13.336089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" event={"ID":"86e394c9-4229-4cf1-b75f-98672d79b8d1","Type":"ContainerStarted","Data":"91d741d8be768c747da0ff658ca5bb55ad6155981e8278eb61a69daf894abc2d"} Jul 08 17:43:13.698965 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:13.698929 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:13.799545 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:13.799449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:16.353162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:16.352999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" event={"ID":"86e394c9-4229-4cf1-b75f-98672d79b8d1","Type":"ContainerStarted","Data":"f8f1b2890ebaf19a623aaf706722c7e54e3d5b9fba572e7452b23f4eeaf5ca46"} Jul 08 17:43:17.358464 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:17.358430 2578 generic.go:356] "Generic (PLEG): container finished" podID="86e394c9-4229-4cf1-b75f-98672d79b8d1" containerID="f8f1b2890ebaf19a623aaf706722c7e54e3d5b9fba572e7452b23f4eeaf5ca46" exitCode=0 Jul 08 17:43:17.358859 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:17.358551 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" event={"ID":"86e394c9-4229-4cf1-b75f-98672d79b8d1","Type":"ContainerDied","Data":"f8f1b2890ebaf19a623aaf706722c7e54e3d5b9fba572e7452b23f4eeaf5ca46"} Jul 08 17:43:18.364017 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:18.363975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" event={"ID":"86e394c9-4229-4cf1-b75f-98672d79b8d1","Type":"ContainerStarted","Data":"8b0d39490cbc54140db3663279f9e8d00cdedc94d8ed77bec739889af0fc615f"} Jul 08 17:43:18.364407 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:18.364120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:18.386247 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:18.386191 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" podStartSLOduration=1.890455193 podStartE2EDuration="6.386173065s" totalImagesPullingTime="3.456688814s" totalInitContainerRuntime="1.039029058s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:18.385057475 +0000 UTC m=+574.682700642" watchObservedRunningTime="2026-07-08 17:43:18.386173065 +0000 UTC m=+574.683816234" Jul 08 17:43:23.600113 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:23.600074 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:23.721125 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:23.721088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:28.287924 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:28.287865 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:28.389971 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:28.389933 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:29.370187 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:29.370158 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c548d5447-7plc5" Jul 08 17:43:30.285887 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.285854 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n"] Jul 08 17:43:30.292854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.292831 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.296146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.296124 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 17:43:30.296786 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.296768 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-c9x66\"" Jul 08 17:43:30.296908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.296890 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 17:43:30.297934 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.297920 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 17:43:30.309775 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.309754 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n"] Jul 08 17:43:30.431234 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.431198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rdjf\" (UniqueName: \"kubernetes.io/projected/dcf13efa-bc07-4270-b5eb-ea81203fa514-kube-api-access-2rdjf\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.431630 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.431256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.431630 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.431328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-metrics-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.431630 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.431376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dcf13efa-bc07-4270-b5eb-ea81203fa514-manager-config\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.532729 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.532697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dcf13efa-bc07-4270-b5eb-ea81203fa514-manager-config\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.532919 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.532752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rdjf\" (UniqueName: \"kubernetes.io/projected/dcf13efa-bc07-4270-b5eb-ea81203fa514-kube-api-access-2rdjf\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.532919 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.532797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.532919 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.532826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-metrics-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.533568 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.533544 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dcf13efa-bc07-4270-b5eb-ea81203fa514-manager-config\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.535442 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.535421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.535619 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.535599 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dcf13efa-bc07-4270-b5eb-ea81203fa514-metrics-cert\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.541300 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.541242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rdjf\" (UniqueName: \"kubernetes.io/projected/dcf13efa-bc07-4270-b5eb-ea81203fa514-kube-api-access-2rdjf\") pod \"lws-controller-manager-8d6bb58cc-gg69n\" (UID: \"dcf13efa-bc07-4270-b5eb-ea81203fa514\") " pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.602202 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.602174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:30.742042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:30.742019 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n"] Jul 08 17:43:30.744320 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:43:30.744293 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf13efa_bc07_4270_b5eb_ea81203fa514.slice/crio-2b8852354f5706f1a012fb289771d4ce8f6d7a86c123a30182b3b1269af94210 WatchSource:0}: Error finding container 2b8852354f5706f1a012fb289771d4ce8f6d7a86c123a30182b3b1269af94210: Status 404 returned error can't find the container with id 2b8852354f5706f1a012fb289771d4ce8f6d7a86c123a30182b3b1269af94210 Jul 08 17:43:31.413684 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:31.413652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" event={"ID":"dcf13efa-bc07-4270-b5eb-ea81203fa514","Type":"ContainerStarted","Data":"2b8852354f5706f1a012fb289771d4ce8f6d7a86c123a30182b3b1269af94210"} Jul 08 17:43:32.086991 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:32.086955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:32.245172 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:32.245076 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:32.705096 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:32.705054 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:32.974529 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:32.974477 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.132583 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.132543 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-bxhp7"] Jul 08 17:43:33.136188 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.136166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.140254 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.140218 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 17:43:33.140596 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.140572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 17:43:33.141187 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.141168 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w6l9g\"" Jul 08 17:43:33.141187 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.141176 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 17:43:33.141340 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.141175 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 17:43:33.154735 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.154708 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-bxhp7"] Jul 08 17:43:33.256130 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.256041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39631309-4184-4f24-a455-55760b1c12f0-tls-certs\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.256130 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.256120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjs2m\" (UniqueName: \"kubernetes.io/projected/39631309-4184-4f24-a455-55760b1c12f0-kube-api-access-cjs2m\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.256311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.256154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39631309-4184-4f24-a455-55760b1c12f0-tmp\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.357570 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.357532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjs2m\" (UniqueName: \"kubernetes.io/projected/39631309-4184-4f24-a455-55760b1c12f0-kube-api-access-cjs2m\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.357733 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.357590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39631309-4184-4f24-a455-55760b1c12f0-tmp\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.357733 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.357632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39631309-4184-4f24-a455-55760b1c12f0-tls-certs\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.360109 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.360074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39631309-4184-4f24-a455-55760b1c12f0-tmp\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.360371 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.360351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39631309-4184-4f24-a455-55760b1c12f0-tls-certs\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.375630 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.375600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjs2m\" (UniqueName: \"kubernetes.io/projected/39631309-4184-4f24-a455-55760b1c12f0-kube-api-access-cjs2m\") pod \"kube-auth-proxy-6674766f74-bxhp7\" (UID: \"39631309-4184-4f24-a455-55760b1c12f0\") " pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.425495 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.425459 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" event={"ID":"dcf13efa-bc07-4270-b5eb-ea81203fa514","Type":"ContainerStarted","Data":"3c28ef38e4d4d33dc776e4996e8be6963fed8c639aafd7bdda225e07cf0f7a09"} Jul 08 17:43:33.425678 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.425637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:33.446819 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.446785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" Jul 08 17:43:33.482693 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.482630 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" podStartSLOduration=1.755256332 podStartE2EDuration="3.482605555s" totalImagesPullingTime="1.727349223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:33.4814041 +0000 UTC m=+589.779047259" watchObservedRunningTime="2026-07-08 17:43:33.482605555 +0000 UTC m=+589.780248723" Jul 08 17:43:33.489444 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.489359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.598778 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.598746 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6674766f74-bxhp7"] Jul 08 17:43:33.602543 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.602493 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.604692 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:43:33.604668 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39631309_4184_4f24_a455_55760b1c12f0.slice/crio-1d06eaa73fc99332f81e0e4d037822300706ae1c93da0c7391613038afe3cef8 WatchSource:0}: Error finding container 1d06eaa73fc99332f81e0e4d037822300706ae1c93da0c7391613038afe3cef8: Status 404 returned error can't find the container with id 1d06eaa73fc99332f81e0e4d037822300706ae1c93da0c7391613038afe3cef8 Jul 08 17:43:33.711663 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.711629 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.811503 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.811421 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.946639 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:33.946604 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:34.396058 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:34.396019 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:34.431296 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:34.431257 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" event={"ID":"39631309-4184-4f24-a455-55760b1c12f0","Type":"ContainerStarted","Data":"1d06eaa73fc99332f81e0e4d037822300706ae1c93da0c7391613038afe3cef8"} Jul 08 17:43:34.717812 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:34.717725 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:35.042139 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:35.042063 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:35.147607 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:35.147576 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:36.417195 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:36.417119 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:37.445311 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:37.445269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" event={"ID":"39631309-4184-4f24-a455-55760b1c12f0","Type":"ContainerStarted","Data":"3574f9777cb9cf42ecd336cb18b4ac239068766a6456a1b8c056a8d88b68a585"} Jul 08 17:43:37.472414 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:37.472366 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6674766f74-bxhp7" podStartSLOduration=1.347893741 podStartE2EDuration="4.472350476s" totalImagesPullingTime="3.124456735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:37.471032663 +0000 UTC m=+593.768675832" watchObservedRunningTime="2026-07-08 17:43:37.472350476 +0000 UTC m=+593.769993643" Jul 08 17:43:41.248604 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.248568 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:41.474688 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.474654 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:41.732125 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.732090 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz"] Jul 08 17:43:41.736397 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.736371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.739756 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.739734 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pw79g\"" Jul 08 17:43:41.740232 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.740144 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 17:43:41.752208 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.752176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz"] Jul 08 17:43:41.788585 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.788506 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:41.834753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797a9d1b-747d-4441-9d58-fb0d0267e848-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.834950 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64s6\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-kube-api-access-w64s6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.835221 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.834974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64s6\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-kube-api-access-w64s6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936502 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936794 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936794 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797a9d1b-747d-4441-9d58-fb0d0267e848-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936794 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936887 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936930 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.936994 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.936972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.937252 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.937236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797a9d1b-747d-4441-9d58-fb0d0267e848-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.939030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.939004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.939030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.939018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.950240 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.950206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:41.950343 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:41.950265 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64s6\" (UniqueName: \"kubernetes.io/projected/797a9d1b-747d-4441-9d58-fb0d0267e848-kube-api-access-w64s6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4lbwmz\" (UID: \"797a9d1b-747d-4441-9d58-fb0d0267e848\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:42.049598 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:42.049480 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:42.054850 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:42.054810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:42.134104 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:42.134072 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:42.206636 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:42.206607 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz"] Jul 08 17:43:42.208045 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:43:42.208022 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797a9d1b_747d_4441_9d58_fb0d0267e848.slice/crio-cd20cf446f6f474cc7ece71126f04f4974d180003bda846f338bcbd5a16cbae7 WatchSource:0}: Error finding container cd20cf446f6f474cc7ece71126f04f4974d180003bda846f338bcbd5a16cbae7: Status 404 returned error can't find the container with id cd20cf446f6f474cc7ece71126f04f4974d180003bda846f338bcbd5a16cbae7 Jul 08 17:43:42.465525 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:42.465492 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" event={"ID":"797a9d1b-747d-4441-9d58-fb0d0267e848","Type":"ContainerStarted","Data":"cd20cf446f6f474cc7ece71126f04f4974d180003bda846f338bcbd5a16cbae7"} Jul 08 17:43:43.797077 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:43.797040 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:43.916678 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:43.916646 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:44.033366 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:44.033310 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:44.122928 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:44.122900 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:44.290181 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:44.290144 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:44.434217 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:44.434141 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8d6bb58cc-gg69n" Jul 08 17:43:44.883193 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:44.883164 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:45.306837 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:45.306799 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:43:45.306924 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:45.306877 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:43:45.306924 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:45.306912 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:43:45.483205 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:45.483169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" event={"ID":"797a9d1b-747d-4441-9d58-fb0d0267e848","Type":"ContainerStarted","Data":"893e739a3aab137b42d94b9a43239c4d13ac528c48b4b9cf9dbd47b5b5e434de"} Jul 08 17:43:45.516564 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:45.516478 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" podStartSLOduration=1.419886972 podStartE2EDuration="4.516458177s" totalImagesPullingTime="3.096571205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:45.514955547 +0000 UTC m=+601.812598715" watchObservedRunningTime="2026-07-08 17:43:45.516458177 +0000 UTC m=+601.814101347" Jul 08 17:43:46.050145 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:46.050111 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:46.054941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:46.054921 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:46.488032 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:46.487996 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:43:46.489142 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:43:46.489124 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4lbwmz" Jul 08 17:44:04.763596 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:44:04.763498 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:08.992324 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:44:08.992285 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:33.595080 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:44:33.595036 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:33.697343 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:44:33.697309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:34.188567 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:44:34.188495 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:01.392404 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:01.392372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:03.040534 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.040485 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f66b4b69-9fthx"] Jul 08 17:45:03.044477 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.044451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.058685 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.058662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f66b4b69-9fthx"] Jul 08 17:45:03.129109 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wqnd\" (UniqueName: \"kubernetes.io/projected/01844b12-7984-4ed5-8779-ee3b94d38e18-kube-api-access-8wqnd\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-oauth-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129243 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-service-ca\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-oauth-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-console-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.129344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.129319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-trusted-ca-bundle\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.229843 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-oauth-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.229843 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-console-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230071 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-trusted-ca-bundle\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230071 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wqnd\" (UniqueName: \"kubernetes.io/projected/01844b12-7984-4ed5-8779-ee3b94d38e18-kube-api-access-8wqnd\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230071 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-oauth-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230071 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230071 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.229971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-service-ca\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230691 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.230664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-service-ca\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230803 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.230697 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-console-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.230803 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.230675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-oauth-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.231073 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.231051 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01844b12-7984-4ed5-8779-ee3b94d38e18-trusted-ca-bundle\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.232621 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.232599 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-oauth-config\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.232773 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.232753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/01844b12-7984-4ed5-8779-ee3b94d38e18-console-serving-cert\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.241259 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.241236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wqnd\" (UniqueName: \"kubernetes.io/projected/01844b12-7984-4ed5-8779-ee3b94d38e18-kube-api-access-8wqnd\") pod \"console-68f66b4b69-9fthx\" (UID: \"01844b12-7984-4ed5-8779-ee3b94d38e18\") " pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.355497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.355464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:03.487110 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.487088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f66b4b69-9fthx"] Jul 08 17:45:03.489354 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:45:03.489330 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01844b12_7984_4ed5_8779_ee3b94d38e18.slice/crio-8cf28293853be243891a7fa4efe1ce8a7745aeb386b2c9143fe6932720404005 WatchSource:0}: Error finding container 8cf28293853be243891a7fa4efe1ce8a7745aeb386b2c9143fe6932720404005: Status 404 returned error can't find the container with id 8cf28293853be243891a7fa4efe1ce8a7745aeb386b2c9143fe6932720404005 Jul 08 17:45:03.803294 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.803202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f66b4b69-9fthx" event={"ID":"01844b12-7984-4ed5-8779-ee3b94d38e18","Type":"ContainerStarted","Data":"476ff5a449d630a6167acf95f95dbfb180e863810ec71ff4a82bc3424b5ec023"} Jul 08 17:45:03.803294 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.803256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f66b4b69-9fthx" event={"ID":"01844b12-7984-4ed5-8779-ee3b94d38e18","Type":"ContainerStarted","Data":"8cf28293853be243891a7fa4efe1ce8a7745aeb386b2c9143fe6932720404005"} Jul 08 17:45:03.842628 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:03.842575 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f66b4b69-9fthx" podStartSLOduration=0.842559813 podStartE2EDuration="842.559813ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:03.841012684 +0000 UTC m=+680.138655852" watchObservedRunningTime="2026-07-08 17:45:03.842559813 +0000 UTC m=+680.140202980" Jul 08 17:45:04.291893 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:04.291853 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:07.562083 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:07.562047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:09.759443 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:09.759402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:10.090212 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:10.090104 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:11.238847 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:11.238768 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:13.355899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.355869 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:13.356266 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.355930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:13.360613 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.360594 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:13.569349 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.569315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:13.664360 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.664275 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:13.853324 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.853296 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f66b4b69-9fthx" Jul 08 17:45:13.974785 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:13.974697 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:45:14.142320 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:14.142266 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:14.244226 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:14.244116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:14.341784 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:14.341750 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:14.441662 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:14.441635 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:15.345081 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.345046 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc"] Jul 08 17:45:15.348180 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.348163 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:15.350694 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.350671 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 17:45:15.351424 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.351406 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 17:45:15.351424 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.351421 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x5gh4\"" Jul 08 17:45:15.351602 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.351408 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 17:45:15.365480 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.365449 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc"] Jul 08 17:45:15.536993 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.536955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xz6q\" (UniqueName: \"kubernetes.io/projected/e4e924fd-64bc-4a84-abd1-44b25ad068e8-kube-api-access-7xz6q\") pod \"dns-operator-controller-manager-864dd78c4c-rc9bc\" (UID: \"e4e924fd-64bc-4a84-abd1-44b25ad068e8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:15.638231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.638191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xz6q\" (UniqueName: \"kubernetes.io/projected/e4e924fd-64bc-4a84-abd1-44b25ad068e8-kube-api-access-7xz6q\") pod \"dns-operator-controller-manager-864dd78c4c-rc9bc\" (UID: \"e4e924fd-64bc-4a84-abd1-44b25ad068e8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:15.648207 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.648176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xz6q\" (UniqueName: \"kubernetes.io/projected/e4e924fd-64bc-4a84-abd1-44b25ad068e8-kube-api-access-7xz6q\") pod \"dns-operator-controller-manager-864dd78c4c-rc9bc\" (UID: \"e4e924fd-64bc-4a84-abd1-44b25ad068e8\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:15.657988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.657966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:15.808421 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.808392 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc"] Jul 08 17:45:15.810159 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:45:15.810138 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e924fd_64bc_4a84_abd1_44b25ad068e8.slice/crio-0345cdbda16f1401b876dc8d11ab8e2e92c4956e9606601c8e7f6bba481d369f WatchSource:0}: Error finding container 0345cdbda16f1401b876dc8d11ab8e2e92c4956e9606601c8e7f6bba481d369f: Status 404 returned error can't find the container with id 0345cdbda16f1401b876dc8d11ab8e2e92c4956e9606601c8e7f6bba481d369f Jul 08 17:45:15.812261 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.812246 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:45:15.859104 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:15.859074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" event={"ID":"e4e924fd-64bc-4a84-abd1-44b25ad068e8","Type":"ContainerStarted","Data":"0345cdbda16f1401b876dc8d11ab8e2e92c4956e9606601c8e7f6bba481d369f"} Jul 08 17:45:16.821535 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:16.821493 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:17.104485 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:17.104449 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:17.215782 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:17.215751 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:17.294126 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:17.294093 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:19.881273 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:19.881232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" event={"ID":"e4e924fd-64bc-4a84-abd1-44b25ad068e8","Type":"ContainerStarted","Data":"7646de204715381c10fd48ad93cc0c68e26712b0421adb93a89af0d727a315fa"} Jul 08 17:45:19.881657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:19.881381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:19.902876 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:19.902824 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" podStartSLOduration=1.490709417 podStartE2EDuration="4.902806657s" totalImagesPullingTime="3.41209724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:19.900748981 +0000 UTC m=+696.198392148" watchObservedRunningTime="2026-07-08 17:45:19.902806657 +0000 UTC m=+696.200449824" Jul 08 17:45:20.957133 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:20.957077 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:21.066314 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:21.066278 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:22.800820 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.800789 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6"] Jul 08 17:45:22.804041 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.804021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:22.806758 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.806735 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 17:45:22.806758 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.806747 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 17:45:22.806961 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.806840 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slgqj\"" Jul 08 17:45:22.813327 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.813307 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6"] Jul 08 17:45:22.816369 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.816344 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:22.904860 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.904821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zphcs\" (UniqueName: \"kubernetes.io/projected/1ec16576-b065-4b66-b935-592c58ffe0fa-kube-api-access-zphcs\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:22.905001 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.904865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ec16576-b065-4b66-b935-592c58ffe0fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:22.905001 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:22.904889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.005863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.005829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zphcs\" (UniqueName: \"kubernetes.io/projected/1ec16576-b065-4b66-b935-592c58ffe0fa-kube-api-access-zphcs\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.005863 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.005870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ec16576-b065-4b66-b935-592c58ffe0fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.006116 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.005894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.006116 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:45:23.006024 2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 08 17:45:23.006116 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:45:23.006081 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert podName:1ec16576-b065-4b66-b935-592c58ffe0fa nodeName:}" failed. No retries permitted until 2026-07-08 17:45:23.506064488 +0000 UTC m=+699.803707648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-q9tj6" (UID: "1ec16576-b065-4b66-b935-592c58ffe0fa") : secret "plugin-serving-cert" not found Jul 08 17:45:23.006587 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.006567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1ec16576-b065-4b66-b935-592c58ffe0fa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.018616 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.018584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zphcs\" (UniqueName: \"kubernetes.io/projected/1ec16576-b065-4b66-b935-592c58ffe0fa-kube-api-access-zphcs\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.265416 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.265377 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:23.510876 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.510841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.513398 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.513357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1ec16576-b065-4b66-b935-592c58ffe0fa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q9tj6\" (UID: \"1ec16576-b065-4b66-b935-592c58ffe0fa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.606787 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.606743 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:23.712454 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.712419 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:23.713232 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.713216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" Jul 08 17:45:23.840289 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.840266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6"] Jul 08 17:45:23.843414 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:45:23.843389 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ec16576_b065_4b66_b935_592c58ffe0fa.slice/crio-0861e3ff270507d0e628b0b31034a0fbe05c4e4f6118b1c8ad9a86c9c9f43aa1 WatchSource:0}: Error finding container 0861e3ff270507d0e628b0b31034a0fbe05c4e4f6118b1c8ad9a86c9c9f43aa1: Status 404 returned error can't find the container with id 0861e3ff270507d0e628b0b31034a0fbe05c4e4f6118b1c8ad9a86c9c9f43aa1 Jul 08 17:45:23.898669 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:23.898585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" event={"ID":"1ec16576-b065-4b66-b935-592c58ffe0fa","Type":"ContainerStarted","Data":"0861e3ff270507d0e628b0b31034a0fbe05c4e4f6118b1c8ad9a86c9c9f43aa1"} Jul 08 17:45:24.358226 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:24.358192 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:24.426830 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:24.426759 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:27.072147 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:27.072111 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:27.172384 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:27.172347 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:30.607548 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:30.607491 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:30.706935 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:30.706893 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:30.887920 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:30.887827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rc9bc" Jul 08 17:45:36.162103 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:36.162013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:36.262048 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:36.262011 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:36.764375 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:36.764342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:37.293487 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:37.293456 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:38.995493 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:38.995452 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5fb4d5787d-g4cvq" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" containerName="console" containerID="cri-o://3b6224ba5e388fba7236b4dfbc2a5daf185a963578223bada57d1edc1cdbaabf" gracePeriod=15 Jul 08 17:45:40.178449 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:40.178399 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:40.278498 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:40.278452 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:41.006152 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:41.006115 2578 patch_prober.go:28] interesting pod/console-5fb4d5787d-g4cvq container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 08 17:45:41.006366 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:41.006186 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5fb4d5787d-g4cvq" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 08 17:45:43.986938 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:43.986897 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:44.056726 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:44.056689 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:49.549906 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:49.549874 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:49.655360 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:49.655330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:50.024844 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.024814 2578 generic.go:356] "Generic (PLEG): container finished" podID="38702120-936a-4f53-ad69-3b30289d5eaa" containerID="3b6224ba5e388fba7236b4dfbc2a5daf185a963578223bada57d1edc1cdbaabf" exitCode=0 Jul 08 17:45:50.024986 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.024890 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4d5787d-g4cvq" event={"ID":"38702120-936a-4f53-ad69-3b30289d5eaa","Type":"ContainerDied","Data":"3b6224ba5e388fba7236b4dfbc2a5daf185a963578223bada57d1edc1cdbaabf"} Jul 08 17:45:50.067001 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.066977 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:45:50.165753 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165727 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.165847 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165772 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.165847 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165805 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.165847 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165841 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.165988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfpcp\" (UniqueName: \"kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.165988 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165915 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.166091 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.165986 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config\") pod \"38702120-936a-4f53-ad69-3b30289d5eaa\" (UID: \"38702120-936a-4f53-ad69-3b30289d5eaa\") " Jul 08 17:45:50.166275 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.166236 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca" (OuterVolumeSpecName: "service-ca") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:45:50.166390 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.166288 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:45:50.166390 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.166297 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config" (OuterVolumeSpecName: "console-config") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:45:50.166390 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.166323 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:45:50.168424 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.168396 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp" (OuterVolumeSpecName: "kube-api-access-sfpcp") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "kube-api-access-sfpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:50.168634 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.168613 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:45:50.168706 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.168637 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38702120-936a-4f53-ad69-3b30289d5eaa" (UID: "38702120-936a-4f53-ad69-3b30289d5eaa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:45:50.266957 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266933 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-console-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.266957 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266957 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpcp\" (UniqueName: \"kubernetes.io/projected/38702120-936a-4f53-ad69-3b30289d5eaa-kube-api-access-sfpcp\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.267124 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266967 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-oauth-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.267124 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266976 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-oauth-config\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.267124 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266986 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.267124 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.266994 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38702120-936a-4f53-ad69-3b30289d5eaa-trusted-ca-bundle\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:50.267124 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:50.267003 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38702120-936a-4f53-ad69-3b30289d5eaa-console-serving-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:45:51.029891 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.029854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" event={"ID":"1ec16576-b065-4b66-b935-592c58ffe0fa","Type":"ContainerStarted","Data":"31a8d3be45f5d37dd69b01ea7fdfa61d51a7b6f3e6a3c69ab6ffdeedee27d271"} Jul 08 17:45:51.031162 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.031136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fb4d5787d-g4cvq" event={"ID":"38702120-936a-4f53-ad69-3b30289d5eaa","Type":"ContainerDied","Data":"6faf6398dce3186c02f82570ce92f03d6ed188ad76a9cb003f8f6a271db17ffa"} Jul 08 17:45:51.031274 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.031164 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fb4d5787d-g4cvq" Jul 08 17:45:51.031274 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.031183 2578 scope.go:117] "RemoveContainer" containerID="3b6224ba5e388fba7236b4dfbc2a5daf185a963578223bada57d1edc1cdbaabf" Jul 08 17:45:51.046476 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.046433 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q9tj6" podStartSLOduration=2.770848301 podStartE2EDuration="29.046415716s" totalImagesPullingTime="26.275567415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:51.044963749 +0000 UTC m=+727.342606916" watchObservedRunningTime="2026-07-08 17:45:51.046415716 +0000 UTC m=+727.344058882" Jul 08 17:45:51.059541 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.059496 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:45:51.065574 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:51.065550 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fb4d5787d-g4cvq"] Jul 08 17:45:52.309293 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:45:52.309254 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" path="/var/lib/kubelet/pods/38702120-936a-4f53-ad69-3b30289d5eaa/volumes" Jul 08 17:46:02.614584 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.614552 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f"] Jul 08 17:46:02.615012 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.614859 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" containerName="console" Jul 08 17:46:02.615012 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.614870 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" containerName="console" Jul 08 17:46:02.615012 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.614934 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38702120-936a-4f53-ad69-3b30289d5eaa" containerName="console" Jul 08 17:46:02.640804 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.640768 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:02.646867 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.646843 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f"] Jul 08 17:46:02.647007 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.646954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.649432 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.649410 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hcj\"" Jul 08 17:46:02.671181 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671252 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8l9\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-kube-api-access-xn8l9\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671275 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671634 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.671634 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.671443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772037 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772037 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772307 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8l9\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-kube-api-access-xn8l9\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772470 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772563 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772771 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.772983 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.773032 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.772990 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.774761 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.774731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.774859 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.774792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.781394 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.781371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.781581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.781561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8l9\" (UniqueName: \"kubernetes.io/projected/7bd20f55-ae1e-4ab1-8848-4b001f3c4883-kube-api-access-xn8l9\") pod \"maas-default-gateway-openshift-default-56654dcf7-p7t6f\" (UID: \"7bd20f55-ae1e-4ab1-8848-4b001f3c4883\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:02.883471 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.883386 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:02.956970 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:02.956937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:03.096448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:03.096425 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f"] Jul 08 17:46:03.099116 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:03.099085 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd20f55_ae1e_4ab1_8848_4b001f3c4883.slice/crio-a7c0accd991ac42b5d3d4df96c9123d70342da39c8b3230ced328edc75755340 WatchSource:0}: Error finding container a7c0accd991ac42b5d3d4df96c9123d70342da39c8b3230ced328edc75755340: Status 404 returned error can't find the container with id a7c0accd991ac42b5d3d4df96c9123d70342da39c8b3230ced328edc75755340 Jul 08 17:46:03.101386 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:03.101355 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:46:03.101448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:03.101424 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:46:03.101485 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:03.101451 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:46:04.093931 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:04.093896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" event={"ID":"7bd20f55-ae1e-4ab1-8848-4b001f3c4883","Type":"ContainerStarted","Data":"68d340e7ccadb6beaac19f4f04419e78d8a3c7c505a679e3dcf3c7756ec0349e"} Jul 08 17:46:04.094332 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:04.093953 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" event={"ID":"7bd20f55-ae1e-4ab1-8848-4b001f3c4883","Type":"ContainerStarted","Data":"a7c0accd991ac42b5d3d4df96c9123d70342da39c8b3230ced328edc75755340"} Jul 08 17:46:04.116460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:04.116378 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" podStartSLOduration=2.116359083 podStartE2EDuration="2.116359083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:04.115155332 +0000 UTC m=+740.412798499" watchObservedRunningTime="2026-07-08 17:46:04.116359083 +0000 UTC m=+740.414002250" Jul 08 17:46:04.957705 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:04.957660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:04.962493 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:04.962466 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:05.097683 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:05.097651 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:05.098783 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:05.098761 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:46:15.223856 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:15.223823 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:15.405187 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:15.405157 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:15.506822 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:15.506742 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:16.090001 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.089968 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:16.128824 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.128791 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:16.150142 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.150088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:16.150318 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.150199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:16.154003 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.153980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-djkd2\"" Jul 08 17:46:16.186155 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.186116 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkpz\" (UniqueName: \"kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz\") pod \"authorino-58cd955675-8m2s7\" (UID: \"b513ebf7-aa75-4a35-b8c9-e8923a2d5462\") " pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:16.233290 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.233239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:16.274149 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.274118 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:16.284158 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.284126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:16.286471 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.286441 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:16.286898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.286872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkpz\" (UniqueName: \"kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz\") pod \"authorino-58cd955675-8m2s7\" (UID: \"b513ebf7-aa75-4a35-b8c9-e8923a2d5462\") " pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:16.295257 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.295233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkpz\" (UniqueName: \"kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz\") pod \"authorino-58cd955675-8m2s7\" (UID: \"b513ebf7-aa75-4a35-b8c9-e8923a2d5462\") " pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:16.387906 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.387870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wzc\" (UniqueName: \"kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc\") pod \"authorino-75d556bd77-p6g4h\" (UID: \"f444fa2a-8c7c-4073-8503-d7245e8ba690\") " pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:16.459688 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.459654 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:16.488820 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.488788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wzc\" (UniqueName: \"kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc\") pod \"authorino-75d556bd77-p6g4h\" (UID: \"f444fa2a-8c7c-4073-8503-d7245e8ba690\") " pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:16.499705 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.499678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wzc\" (UniqueName: \"kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc\") pod \"authorino-75d556bd77-p6g4h\" (UID: \"f444fa2a-8c7c-4073-8503-d7245e8ba690\") " pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:16.590658 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.590536 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:16.593202 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:16.593173 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb513ebf7_aa75_4a35_b8c9_e8923a2d5462.slice/crio-b066aa07ce2481206ec3e119989adbd8c1fea89ec7b57eb4abae0a55ec1fe026 WatchSource:0}: Error finding container b066aa07ce2481206ec3e119989adbd8c1fea89ec7b57eb4abae0a55ec1fe026: Status 404 returned error can't find the container with id b066aa07ce2481206ec3e119989adbd8c1fea89ec7b57eb4abae0a55ec1fe026 Jul 08 17:46:16.593910 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.593888 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:16.738716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:16.738691 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:16.741009 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:16.740969 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf444fa2a_8c7c_4073_8503_d7245e8ba690.slice/crio-e3a0017bf84bc378d37dace72742f237e930207025ae3ce07a0cb55085b2792b WatchSource:0}: Error finding container e3a0017bf84bc378d37dace72742f237e930207025ae3ce07a0cb55085b2792b: Status 404 returned error can't find the container with id e3a0017bf84bc378d37dace72742f237e930207025ae3ce07a0cb55085b2792b Jul 08 17:46:17.155116 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:17.155077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p6g4h" event={"ID":"f444fa2a-8c7c-4073-8503-d7245e8ba690","Type":"ContainerStarted","Data":"e3a0017bf84bc378d37dace72742f237e930207025ae3ce07a0cb55085b2792b"} Jul 08 17:46:17.156064 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:17.156043 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8m2s7" event={"ID":"b513ebf7-aa75-4a35-b8c9-e8923a2d5462","Type":"ContainerStarted","Data":"b066aa07ce2481206ec3e119989adbd8c1fea89ec7b57eb4abae0a55ec1fe026"} Jul 08 17:46:21.178547 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:21.178482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8m2s7" event={"ID":"b513ebf7-aa75-4a35-b8c9-e8923a2d5462","Type":"ContainerStarted","Data":"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c"} Jul 08 17:46:21.179977 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:21.179947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p6g4h" event={"ID":"f444fa2a-8c7c-4073-8503-d7245e8ba690","Type":"ContainerStarted","Data":"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca"} Jul 08 17:46:21.197380 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:21.197332 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8m2s7" podStartSLOduration=1.3934458969999999 podStartE2EDuration="5.197317883s" totalImagesPullingTime="3.803871986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:21.196676402 +0000 UTC m=+757.494319569" watchObservedRunningTime="2026-07-08 17:46:21.197317883 +0000 UTC m=+757.494961067" Jul 08 17:46:21.216102 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:21.216053 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-p6g4h" podStartSLOduration=1.57186335 podStartE2EDuration="5.216037605s" totalImagesPullingTime="3.644174255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:21.21548399 +0000 UTC m=+757.513127158" watchObservedRunningTime="2026-07-08 17:46:21.216037605 +0000 UTC m=+757.513680773" Jul 08 17:46:21.244498 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:21.244467 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:23.189970 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:23.189931 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8m2s7" podUID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" containerName="authorino" containerID="cri-o://e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c" gracePeriod=30 Jul 08 17:46:23.444555 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:23.444474 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:23.559044 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:23.559009 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hkpz\" (UniqueName: \"kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz\") pod \"b513ebf7-aa75-4a35-b8c9-e8923a2d5462\" (UID: \"b513ebf7-aa75-4a35-b8c9-e8923a2d5462\") " Jul 08 17:46:23.561280 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:23.561251 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz" (OuterVolumeSpecName: "kube-api-access-9hkpz") pod "b513ebf7-aa75-4a35-b8c9-e8923a2d5462" (UID: "b513ebf7-aa75-4a35-b8c9-e8923a2d5462"). InnerVolumeSpecName "kube-api-access-9hkpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:23.659737 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:23.659702 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkpz\" (UniqueName: \"kubernetes.io/projected/b513ebf7-aa75-4a35-b8c9-e8923a2d5462-kube-api-access-9hkpz\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:46:24.195306 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.195273 2578 generic.go:356] "Generic (PLEG): container finished" podID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" containerID="e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c" exitCode=0 Jul 08 17:46:24.195808 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.195323 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8m2s7" Jul 08 17:46:24.195808 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.195367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8m2s7" event={"ID":"b513ebf7-aa75-4a35-b8c9-e8923a2d5462","Type":"ContainerDied","Data":"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c"} Jul 08 17:46:24.195808 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.195411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8m2s7" event={"ID":"b513ebf7-aa75-4a35-b8c9-e8923a2d5462","Type":"ContainerDied","Data":"b066aa07ce2481206ec3e119989adbd8c1fea89ec7b57eb4abae0a55ec1fe026"} Jul 08 17:46:24.195808 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.195431 2578 scope.go:117] "RemoveContainer" containerID="e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c" Jul 08 17:46:24.203694 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.203679 2578 scope.go:117] "RemoveContainer" containerID="e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c" Jul 08 17:46:24.203932 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:46:24.203917 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c\": container with ID starting with e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c not found: ID does not exist" containerID="e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c" Jul 08 17:46:24.203975 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.203939 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c"} err="failed to get container status \"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c\": rpc error: code = NotFound desc = could not find container \"e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c\": container with ID starting with e4d4c03587f75e360b65f8a8dc73e702b3797503574fba34e8a62203e3c02f6c not found: ID does not exist" Jul 08 17:46:24.215241 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.215221 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:24.219226 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.219205 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8m2s7"] Jul 08 17:46:24.308833 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:24.308795 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" path="/var/lib/kubelet/pods/b513ebf7-aa75-4a35-b8c9-e8923a2d5462/volumes" Jul 08 17:46:31.596485 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:31.596445 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:32.347941 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:32.347905 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:32.448195 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:32.448163 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:44.836406 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:44.836372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:44.908971 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:44.908938 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:45.096059 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.095979 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:45.096321 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.096308 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" containerName="authorino" Jul 08 17:46:45.096364 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.096322 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" containerName="authorino" Jul 08 17:46:45.096402 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.096371 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b513ebf7-aa75-4a35-b8c9-e8923a2d5462" containerName="authorino" Jul 08 17:46:45.101090 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.101061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:45.110160 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.110133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:45.147239 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.147205 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:45.156303 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.156272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfl2\" (UniqueName: \"kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2\") pod \"authorino-5d4b67c446-fwwh4\" (UID: \"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a\") " pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:45.257263 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.257221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfl2\" (UniqueName: \"kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2\") pod \"authorino-5d4b67c446-fwwh4\" (UID: \"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a\") " pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:45.266098 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.266060 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfl2\" (UniqueName: \"kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2\") pod \"authorino-5d4b67c446-fwwh4\" (UID: \"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a\") " pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:45.379306 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.379266 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:45.379558 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.379544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:45.520345 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.520321 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:45.522558 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:45.522529 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae8fd7e5_0bf8_4666_9a6b_837cd31acb4a.slice/crio-fab43eeded4323db501338941be31d04ddd3948672e7551651b3515a831673b7 WatchSource:0}: Error finding container fab43eeded4323db501338941be31d04ddd3948672e7551651b3515a831673b7: Status 404 returned error can't find the container with id fab43eeded4323db501338941be31d04ddd3948672e7551651b3515a831673b7 Jul 08 17:46:45.582312 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.582274 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:46:45.586905 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.586879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.589433 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.589412 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 17:46:45.596529 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.596488 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:46:45.659796 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.659708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.659796 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.659749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22b7\" (UniqueName: \"kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.760673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.760630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.760673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.760669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22b7\" (UniqueName: \"kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.763296 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.763274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.769210 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.769182 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22b7\" (UniqueName: \"kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7\") pod \"authorino-75c9f44d57-pqgpf\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:45.897885 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:45.897832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:46:46.039891 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:46.039856 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:46:46.042707 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:46.042676 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7bc925_89c5_487c_8409_7646cffcaa7a.slice/crio-5383252c87655315f0b71e606a1a6b78f640d80df386883dd643d3cc9d84a81e WatchSource:0}: Error finding container 5383252c87655315f0b71e606a1a6b78f640d80df386883dd643d3cc9d84a81e: Status 404 returned error can't find the container with id 5383252c87655315f0b71e606a1a6b78f640d80df386883dd643d3cc9d84a81e Jul 08 17:46:46.291821 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:46.291722 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" event={"ID":"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a","Type":"ContainerStarted","Data":"fab43eeded4323db501338941be31d04ddd3948672e7551651b3515a831673b7"} Jul 08 17:46:46.292848 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:46.292824 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" event={"ID":"4b7bc925-89c5-487c-8409-7646cffcaa7a","Type":"ContainerStarted","Data":"5383252c87655315f0b71e606a1a6b78f640d80df386883dd643d3cc9d84a81e"} Jul 08 17:46:47.299951 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.299909 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" event={"ID":"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a","Type":"ContainerStarted","Data":"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e"} Jul 08 17:46:47.299951 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.299921 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" podUID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" containerName="authorino" containerID="cri-o://c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e" gracePeriod=30 Jul 08 17:46:47.301643 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.301613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" event={"ID":"4b7bc925-89c5-487c-8409-7646cffcaa7a","Type":"ContainerStarted","Data":"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff"} Jul 08 17:46:47.318281 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.318222 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" podStartSLOduration=1.130660482 podStartE2EDuration="2.318201944s" totalImagesPullingTime="1.187541462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:47.316285531 +0000 UTC m=+783.613928699" watchObservedRunningTime="2026-07-08 17:46:47.318201944 +0000 UTC m=+783.615845111" Jul 08 17:46:47.334647 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.334586 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" podStartSLOduration=1.862846898 podStartE2EDuration="2.334571114s" totalImagesPullingTime="471.724216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:47.33402253 +0000 UTC m=+783.631665703" watchObservedRunningTime="2026-07-08 17:46:47.334571114 +0000 UTC m=+783.632214282" Jul 08 17:46:47.364390 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.364349 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:47.364606 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.364579 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-p6g4h" podUID="f444fa2a-8c7c-4073-8503-d7245e8ba690" containerName="authorino" containerID="cri-o://6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca" gracePeriod=30 Jul 08 17:46:47.638375 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.638350 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:47.641489 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.641447 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:47.673915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.673872 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpfl2\" (UniqueName: \"kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2\") pod \"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a\" (UID: \"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a\") " Jul 08 17:46:47.674096 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.673924 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5wzc\" (UniqueName: \"kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc\") pod \"f444fa2a-8c7c-4073-8503-d7245e8ba690\" (UID: \"f444fa2a-8c7c-4073-8503-d7245e8ba690\") " Jul 08 17:46:47.676328 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.676297 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2" (OuterVolumeSpecName: "kube-api-access-hpfl2") pod "ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" (UID: "ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a"). InnerVolumeSpecName "kube-api-access-hpfl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:47.676328 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.676313 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc" (OuterVolumeSpecName: "kube-api-access-d5wzc") pod "f444fa2a-8c7c-4073-8503-d7245e8ba690" (UID: "f444fa2a-8c7c-4073-8503-d7245e8ba690"). InnerVolumeSpecName "kube-api-access-d5wzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:47.774740 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.774696 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfl2\" (UniqueName: \"kubernetes.io/projected/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a-kube-api-access-hpfl2\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:46:47.774740 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:47.774733 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wzc\" (UniqueName: \"kubernetes.io/projected/f444fa2a-8c7c-4073-8503-d7245e8ba690-kube-api-access-d5wzc\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:46:48.308032 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.307996 2578 generic.go:356] "Generic (PLEG): container finished" podID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" containerID="c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e" exitCode=0 Jul 08 17:46:48.308476 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.308095 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" Jul 08 17:46:48.309649 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.309606 2578 generic.go:356] "Generic (PLEG): container finished" podID="f444fa2a-8c7c-4073-8503-d7245e8ba690" containerID="6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca" exitCode=0 Jul 08 17:46:48.309766 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.309696 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-p6g4h" Jul 08 17:46:48.309951 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.309928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" event={"ID":"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a","Type":"ContainerDied","Data":"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e"} Jul 08 17:46:48.310042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.309968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-fwwh4" event={"ID":"ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a","Type":"ContainerDied","Data":"fab43eeded4323db501338941be31d04ddd3948672e7551651b3515a831673b7"} Jul 08 17:46:48.310042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.309986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p6g4h" event={"ID":"f444fa2a-8c7c-4073-8503-d7245e8ba690","Type":"ContainerDied","Data":"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca"} Jul 08 17:46:48.310042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.310003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-p6g4h" event={"ID":"f444fa2a-8c7c-4073-8503-d7245e8ba690","Type":"ContainerDied","Data":"e3a0017bf84bc378d37dace72742f237e930207025ae3ce07a0cb55085b2792b"} Jul 08 17:46:48.310042 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.310025 2578 scope.go:117] "RemoveContainer" containerID="c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e" Jul 08 17:46:48.323489 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.323470 2578 scope.go:117] "RemoveContainer" containerID="c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e" Jul 08 17:46:48.323907 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:46:48.323887 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e\": container with ID starting with c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e not found: ID does not exist" containerID="c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e" Jul 08 17:46:48.323981 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.323915 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e"} err="failed to get container status \"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e\": rpc error: code = NotFound desc = could not find container \"c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e\": container with ID starting with c92f803b511076f49d5cc1316a283c619e6ce12804ae6a14ba3475b2a22b553e not found: ID does not exist" Jul 08 17:46:48.323981 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.323932 2578 scope.go:117] "RemoveContainer" containerID="6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca" Jul 08 17:46:48.331902 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.331883 2578 scope.go:117] "RemoveContainer" containerID="6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca" Jul 08 17:46:48.332177 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:46:48.332161 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca\": container with ID starting with 6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca not found: ID does not exist" containerID="6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca" Jul 08 17:46:48.332225 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.332185 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca"} err="failed to get container status \"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca\": rpc error: code = NotFound desc = could not find container \"6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca\": container with ID starting with 6fc743ed76266332eb20696a9f9f6cf11cb3551150e1fa37dbd93a3bbdc764ca not found: ID does not exist" Jul 08 17:46:48.342581 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.342550 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:48.346750 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.346720 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-p6g4h"] Jul 08 17:46:48.359410 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.359375 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:48.364313 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:48.364286 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fwwh4"] Jul 08 17:46:50.309843 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:50.309813 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" path="/var/lib/kubelet/pods/ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a/volumes" Jul 08 17:46:50.310203 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:50.310124 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f444fa2a-8c7c-4073-8503-d7245e8ba690" path="/var/lib/kubelet/pods/f444fa2a-8c7c-4073-8503-d7245e8ba690/volumes" Jul 08 17:46:51.818261 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.818186 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:51.885877 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.885842 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:46:51.886224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886211 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f444fa2a-8c7c-4073-8503-d7245e8ba690" containerName="authorino" Jul 08 17:46:51.886266 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886227 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f444fa2a-8c7c-4073-8503-d7245e8ba690" containerName="authorino" Jul 08 17:46:51.886266 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886243 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" containerName="authorino" Jul 08 17:46:51.886266 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886249 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" containerName="authorino" Jul 08 17:46:51.886381 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886308 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8fd7e5-0bf8-4666-9a6b-837cd31acb4a" containerName="authorino" Jul 08 17:46:51.886381 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.886318 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f444fa2a-8c7c-4073-8503-d7245e8ba690" containerName="authorino" Jul 08 17:46:51.890713 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.890694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:51.893845 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.893819 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6xqmn\"" Jul 08 17:46:51.894108 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.894092 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 17:46:51.894582 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.894564 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 17:46:51.908752 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.908712 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:51.908891 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.908815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nkr\" (UniqueName: \"kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:51.910021 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.909994 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:46:51.986303 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:51.986271 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:52.009727 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.009685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:52.009925 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.009765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nkr\" (UniqueName: \"kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:52.012820 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.012796 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:52.026604 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.026578 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nkr\" (UniqueName: \"kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr\") pod \"maas-controller-684f975fb9-sh95m\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:52.097341 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.097257 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:46:52.100429 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.100410 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.110849 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.110823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.110969 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.110886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpcf\" (UniqueName: \"kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.120395 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.120369 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:46:52.128424 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.128384 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:52.201443 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.201400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:52.211817 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.211795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.211892 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.211846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpcf\" (UniqueName: \"kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.214652 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.214625 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.220974 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.220944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpcf\" (UniqueName: \"kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf\") pod \"maas-controller-67df7499f4-z2rtm\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.340327 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.340292 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:46:52.343761 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:52.343734 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735d0231_4ffa_48f4_985b_2e0ade62ba42.slice/crio-794da8e4d58a1bec73408dcc85be54d434474b640bb90aec2241fc042e8547df WatchSource:0}: Error finding container 794da8e4d58a1bec73408dcc85be54d434474b640bb90aec2241fc042e8547df: Status 404 returned error can't find the container with id 794da8e4d58a1bec73408dcc85be54d434474b640bb90aec2241fc042e8547df Jul 08 17:46:52.411702 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.411662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:52.551693 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:52.551665 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:46:52.554166 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:46:52.554125 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c118e60_d957_4081_837c_37f047041690.slice/crio-e04f87fa148ff3f5b6efd158764766a8efd6c66ad7fbbd3a59c2785dd9e2e4a0 WatchSource:0}: Error finding container e04f87fa148ff3f5b6efd158764766a8efd6c66ad7fbbd3a59c2785dd9e2e4a0: Status 404 returned error can't find the container with id e04f87fa148ff3f5b6efd158764766a8efd6c66ad7fbbd3a59c2785dd9e2e4a0 Jul 08 17:46:53.339374 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:53.339337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-z2rtm" event={"ID":"1c118e60-d957-4081-837c-37f047041690","Type":"ContainerStarted","Data":"e04f87fa148ff3f5b6efd158764766a8efd6c66ad7fbbd3a59c2785dd9e2e4a0"} Jul 08 17:46:53.340947 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:53.340915 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-684f975fb9-sh95m" event={"ID":"735d0231-4ffa-48f4-985b-2e0ade62ba42","Type":"ContainerStarted","Data":"794da8e4d58a1bec73408dcc85be54d434474b640bb90aec2241fc042e8547df"} Jul 08 17:46:55.356211 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.356159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-z2rtm" event={"ID":"1c118e60-d957-4081-837c-37f047041690","Type":"ContainerStarted","Data":"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18"} Jul 08 17:46:55.356676 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.356231 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:46:55.357637 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.357615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-684f975fb9-sh95m" event={"ID":"735d0231-4ffa-48f4-985b-2e0ade62ba42","Type":"ContainerStarted","Data":"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811"} Jul 08 17:46:55.357765 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.357725 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:46:55.376330 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.376286 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67df7499f4-z2rtm" podStartSLOduration=0.673087518 podStartE2EDuration="3.376272333s" totalImagesPullingTime="2.703184815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:55.374268829 +0000 UTC m=+791.671911996" watchObservedRunningTime="2026-07-08 17:46:55.376272333 +0000 UTC m=+791.673915561" Jul 08 17:46:55.392930 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.392880 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-684f975fb9-sh95m" podStartSLOduration=1.483074317 podStartE2EDuration="4.392865428s" totalImagesPullingTime="2.909791111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:55.39196995 +0000 UTC m=+791.689613120" watchObservedRunningTime="2026-07-08 17:46:55.392865428 +0000 UTC m=+791.690508595" Jul 08 17:46:55.485697 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.485660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:55.576339 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:46:55.576254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:06.368731 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.368646 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:47:06.369332 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.369311 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:47:06.430709 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.430678 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:47:06.430923 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.430880 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-684f975fb9-sh95m" podUID="735d0231-4ffa-48f4-985b-2e0ade62ba42" containerName="manager" containerID="cri-o://33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811" gracePeriod=30 Jul 08 17:47:06.675608 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.675580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:47:06.745169 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.745133 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert\") pod \"735d0231-4ffa-48f4-985b-2e0ade62ba42\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " Jul 08 17:47:06.745363 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.745273 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6nkr\" (UniqueName: \"kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr\") pod \"735d0231-4ffa-48f4-985b-2e0ade62ba42\" (UID: \"735d0231-4ffa-48f4-985b-2e0ade62ba42\") " Jul 08 17:47:06.747439 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.747413 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr" (OuterVolumeSpecName: "kube-api-access-b6nkr") pod "735d0231-4ffa-48f4-985b-2e0ade62ba42" (UID: "735d0231-4ffa-48f4-985b-2e0ade62ba42"). InnerVolumeSpecName "kube-api-access-b6nkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:47:06.747535 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.747460 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "735d0231-4ffa-48f4-985b-2e0ade62ba42" (UID: "735d0231-4ffa-48f4-985b-2e0ade62ba42"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:47:06.846346 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.846311 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nkr\" (UniqueName: \"kubernetes.io/projected/735d0231-4ffa-48f4-985b-2e0ade62ba42-kube-api-access-b6nkr\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:47:06.846346 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:06.846344 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/735d0231-4ffa-48f4-985b-2e0ade62ba42-webhook-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:47:07.411921 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.411888 2578 generic.go:356] "Generic (PLEG): container finished" podID="735d0231-4ffa-48f4-985b-2e0ade62ba42" containerID="33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811" exitCode=0 Jul 08 17:47:07.412353 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.411933 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-684f975fb9-sh95m" Jul 08 17:47:07.412353 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.411972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-684f975fb9-sh95m" event={"ID":"735d0231-4ffa-48f4-985b-2e0ade62ba42","Type":"ContainerDied","Data":"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811"} Jul 08 17:47:07.412353 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.412035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-684f975fb9-sh95m" event={"ID":"735d0231-4ffa-48f4-985b-2e0ade62ba42","Type":"ContainerDied","Data":"794da8e4d58a1bec73408dcc85be54d434474b640bb90aec2241fc042e8547df"} Jul 08 17:47:07.412353 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.412070 2578 scope.go:117] "RemoveContainer" containerID="33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811" Jul 08 17:47:07.420649 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.420625 2578 scope.go:117] "RemoveContainer" containerID="33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811" Jul 08 17:47:07.420903 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:47:07.420885 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811\": container with ID starting with 33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811 not found: ID does not exist" containerID="33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811" Jul 08 17:47:07.420964 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.420912 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811"} err="failed to get container status \"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811\": rpc error: code = NotFound desc = could not find container \"33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811\": container with ID starting with 33a85f093547a536713bd5aff3e41f5bfa96b629c0416844bae70d6e55546811 not found: ID does not exist" Jul 08 17:47:07.433580 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.433560 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:47:07.438029 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:07.438011 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-684f975fb9-sh95m"] Jul 08 17:47:08.309992 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:08.309961 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735d0231-4ffa-48f4-985b-2e0ade62ba42" path="/var/lib/kubelet/pods/735d0231-4ffa-48f4-985b-2e0ade62ba42/volumes" Jul 08 17:47:24.496616 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:24.496585 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:25.185592 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.185561 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-798cff898-cbrj5"] Jul 08 17:47:25.185877 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.185865 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735d0231-4ffa-48f4-985b-2e0ade62ba42" containerName="manager" Jul 08 17:47:25.185923 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.185879 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="735d0231-4ffa-48f4-985b-2e0ade62ba42" containerName="manager" Jul 08 17:47:25.185961 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.185946 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="735d0231-4ffa-48f4-985b-2e0ade62ba42" containerName="manager" Jul 08 17:47:25.189249 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.189234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.191602 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.191580 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 17:47:25.191717 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.191591 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 17:47:25.192403 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.192386 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fljv6\"" Jul 08 17:47:25.192453 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.192431 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 17:47:25.199926 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.199907 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798cff898-cbrj5"] Jul 08 17:47:25.221146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.221092 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:25.311359 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.311330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2789f8eb-159d-4959-bf53-122d9f93ac1f-maas-api-tls\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.311496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.311376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6nj\" (UniqueName: \"kubernetes.io/projected/2789f8eb-159d-4959-bf53-122d9f93ac1f-kube-api-access-cv6nj\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.311496 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.311408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2789f8eb-159d-4959-bf53-122d9f93ac1f-openshift-service-ca\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.320934 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.320866 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:25.412746 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.412717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6nj\" (UniqueName: \"kubernetes.io/projected/2789f8eb-159d-4959-bf53-122d9f93ac1f-kube-api-access-cv6nj\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.412908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.412765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2789f8eb-159d-4959-bf53-122d9f93ac1f-openshift-service-ca\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.412908 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.412845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2789f8eb-159d-4959-bf53-122d9f93ac1f-maas-api-tls\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.413436 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.413412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2789f8eb-159d-4959-bf53-122d9f93ac1f-openshift-service-ca\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.415361 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.415343 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2789f8eb-159d-4959-bf53-122d9f93ac1f-maas-api-tls\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.423976 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.423954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6nj\" (UniqueName: \"kubernetes.io/projected/2789f8eb-159d-4959-bf53-122d9f93ac1f-kube-api-access-cv6nj\") pod \"maas-api-798cff898-cbrj5\" (UID: \"2789f8eb-159d-4959-bf53-122d9f93ac1f\") " pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.499270 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.499208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:25.568173 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.568127 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:25.638694 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.638664 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798cff898-cbrj5"] Jul 08 17:47:25.641461 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:47:25.641421 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2789f8eb_159d_4959_bf53_122d9f93ac1f.slice/crio-9fac52cdb09c4f3e9f704235993120fa02067c13b3a566d35e27458d581af621 WatchSource:0}: Error finding container 9fac52cdb09c4f3e9f704235993120fa02067c13b3a566d35e27458d581af621: Status 404 returned error can't find the container with id 9fac52cdb09c4f3e9f704235993120fa02067c13b3a566d35e27458d581af621 Jul 08 17:47:25.642583 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:25.642557 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:26.509673 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:26.509637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" event={"ID":"2789f8eb-159d-4959-bf53-122d9f93ac1f","Type":"ContainerStarted","Data":"9fac52cdb09c4f3e9f704235993120fa02067c13b3a566d35e27458d581af621"} Jul 08 17:47:28.520296 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:28.520261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" event={"ID":"2789f8eb-159d-4959-bf53-122d9f93ac1f","Type":"ContainerStarted","Data":"50ed749a6aa51fedba32d3813404d999c58753e5a243c547ff41507391905289"} Jul 08 17:47:28.520744 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:28.520447 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:28.540181 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:28.540135 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" podStartSLOduration=1.729888476 podStartE2EDuration="3.540119989s" totalImagesPullingTime="1.810231513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:28.539454558 +0000 UTC m=+824.837097727" watchObservedRunningTime="2026-07-08 17:47:28.540119989 +0000 UTC m=+824.837763209" Jul 08 17:47:33.545030 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.544995 2578 generic.go:356] "Generic (PLEG): container finished" podID="7bd20f55-ae1e-4ab1-8848-4b001f3c4883" containerID="68d340e7ccadb6beaac19f4f04419e78d8a3c7c505a679e3dcf3c7756ec0349e" exitCode=0 Jul 08 17:47:33.545457 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.545069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" event={"ID":"7bd20f55-ae1e-4ab1-8848-4b001f3c4883","Type":"ContainerDied","Data":"68d340e7ccadb6beaac19f4f04419e78d8a3c7c505a679e3dcf3c7756ec0349e"} Jul 08 17:47:33.545457 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.545445 2578 scope.go:117] "RemoveContainer" containerID="68d340e7ccadb6beaac19f4f04419e78d8a3c7c505a679e3dcf3c7756ec0349e" Jul 08 17:47:33.546408 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.546384 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:47:33.546460 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.546448 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:47:33.546497 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:33.546476 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 17:47:34.530719 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:34.530693 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-798cff898-cbrj5" Jul 08 17:47:34.551687 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:34.551657 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" event={"ID":"7bd20f55-ae1e-4ab1-8848-4b001f3c4883","Type":"ContainerStarted","Data":"3fe9d255df62416afe74aea464e12aefddda126a37cb567e9e3e4a855f4b82c8"} Jul 08 17:47:34.957829 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:34.957795 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:47:34.959121 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:34.959082 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-p7t6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Jul 08 17:47:34.959264 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:34.959150 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" podUID="7bd20f55-ae1e-4ab1-8848-4b001f3c4883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Jul 08 17:47:35.098125 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:35.098083 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:47:35.574791 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:35.574756 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:35.671157 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:35.671117 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:35.958176 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:35.958141 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-p7t6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Jul 08 17:47:35.958349 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:35.958199 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" podUID="7bd20f55-ae1e-4ab1-8848-4b001f3c4883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Jul 08 17:47:36.958278 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:36.958235 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-p7t6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Jul 08 17:47:36.958990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:36.958762 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" podUID="7bd20f55-ae1e-4ab1-8848-4b001f3c4883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Jul 08 17:47:37.958245 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:37.958208 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-p7t6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Jul 08 17:47:37.958452 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:37.958279 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" podUID="7bd20f55-ae1e-4ab1-8848-4b001f3c4883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Jul 08 17:47:38.962230 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:38.962194 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:47:38.963463 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:38.963439 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-p7t6f" Jul 08 17:47:43.315087 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:43.315039 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:49.481448 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:49.481410 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:56.645580 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.645541 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9"] Jul 08 17:47:56.649290 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.649270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.652793 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.652771 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 08 17:47:56.652945 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.652795 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 17:47:56.652945 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.652836 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 17:47:56.652945 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.652796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zg5mg\"" Jul 08 17:47:56.660697 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.660697 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.660898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660730 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62641589-106e-4c89-b14a-9a810e8128a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.660898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.660898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6g5\" (UniqueName: \"kubernetes.io/projected/62641589-106e-4c89-b14a-9a810e8128a2-kube-api-access-pf6g5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.660898 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.660865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.661826 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.661803 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9"] Jul 08 17:47:56.698696 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.698662 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:56.762265 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762265 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762490 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62641589-106e-4c89-b14a-9a810e8128a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762490 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762490 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6g5\" (UniqueName: \"kubernetes.io/projected/62641589-106e-4c89-b14a-9a810e8128a2-kube-api-access-pf6g5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762754 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.762875 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.763006 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.763006 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.762958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.764815 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.764792 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62641589-106e-4c89-b14a-9a810e8128a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.765195 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.765173 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62641589-106e-4c89-b14a-9a810e8128a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.771716 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.771696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6g5\" (UniqueName: \"kubernetes.io/projected/62641589-106e-4c89-b14a-9a810e8128a2-kube-api-access-pf6g5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-bjbg9\" (UID: \"62641589-106e-4c89-b14a-9a810e8128a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:56.960944 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:56.960830 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:47:57.094137 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:57.094107 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9"] Jul 08 17:47:57.096951 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:47:57.096922 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62641589_106e_4c89_b14a_9a810e8128a2.slice/crio-b8a3fd4acbc72d8363c164725d7d10e9ab6c6be10369eaf0d2f11cef0b94ff95 WatchSource:0}: Error finding container b8a3fd4acbc72d8363c164725d7d10e9ab6c6be10369eaf0d2f11cef0b94ff95: Status 404 returned error can't find the container with id b8a3fd4acbc72d8363c164725d7d10e9ab6c6be10369eaf0d2f11cef0b94ff95 Jul 08 17:47:57.667823 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:57.667786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" event={"ID":"62641589-106e-4c89-b14a-9a810e8128a2","Type":"ContainerStarted","Data":"b8a3fd4acbc72d8363c164725d7d10e9ab6c6be10369eaf0d2f11cef0b94ff95"} Jul 08 17:47:58.674699 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:58.674657 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" event={"ID":"62641589-106e-4c89-b14a-9a810e8128a2","Type":"ContainerStarted","Data":"f42fa2b8dfa973d5f9867186779592c8382e250dd23ff0b96728211526964812"} Jul 08 17:47:58.694777 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:47:58.694724 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" podStartSLOduration=1.6541859429999999 podStartE2EDuration="2.694708844s" totalImagesPullingTime="1.040522901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:58.693246845 +0000 UTC m=+854.990890011" watchObservedRunningTime="2026-07-08 17:47:58.694708844 +0000 UTC m=+854.992352010" Jul 08 17:48:02.863635 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.863598 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz"] Jul 08 17:48:02.867062 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.867044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.869596 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.869575 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 17:48:02.881614 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.881583 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz"] Jul 08 17:48:02.915089 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.915065 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:02.918410 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.918476 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.918476 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.918591 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fc0081-c208-403e-a630-ee4e8f10298e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.918591 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:02.918657 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:02.918607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxpp\" (UniqueName: \"kubernetes.io/projected/52fc0081-c208-403e-a630-ee4e8f10298e-kube-api-access-bbxpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.019719 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fc0081-c208-403e-a630-ee4e8f10298e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.019899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.019899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxpp\" (UniqueName: \"kubernetes.io/projected/52fc0081-c208-403e-a630-ee4e8f10298e-kube-api-access-bbxpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.019899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.019899 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.020116 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.019925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.020209 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.020185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.020446 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.020395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.020644 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.020616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.022833 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.022803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52fc0081-c208-403e-a630-ee4e8f10298e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.023573 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.023373 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52fc0081-c208-403e-a630-ee4e8f10298e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.029336 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.029315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxpp\" (UniqueName: \"kubernetes.io/projected/52fc0081-c208-403e-a630-ee4e8f10298e-kube-api-access-bbxpp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz\" (UID: \"52fc0081-c208-403e-a630-ee4e8f10298e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.177647 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.177570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:03.309641 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.309450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz"] Jul 08 17:48:03.312222 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:48:03.312193 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fc0081_c208_403e_a630_ee4e8f10298e.slice/crio-efc47283bf36cbbf767a9367005de638f112b5f78a7503d5af01ba13cf2df534 WatchSource:0}: Error finding container efc47283bf36cbbf767a9367005de638f112b5f78a7503d5af01ba13cf2df534: Status 404 returned error can't find the container with id efc47283bf36cbbf767a9367005de638f112b5f78a7503d5af01ba13cf2df534 Jul 08 17:48:03.699189 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.699156 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" event={"ID":"52fc0081-c208-403e-a630-ee4e8f10298e","Type":"ContainerStarted","Data":"906d1092610dbcd349f61551a1ab497fac6bf577f05edb7e5da21a53b5a91845"} Jul 08 17:48:03.699354 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.699216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" event={"ID":"52fc0081-c208-403e-a630-ee4e8f10298e","Type":"ContainerStarted","Data":"efc47283bf36cbbf767a9367005de638f112b5f78a7503d5af01ba13cf2df534"} Jul 08 17:48:03.718152 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:03.718099 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" podStartSLOduration=1.442423432 podStartE2EDuration="1.718081601s" totalImagesPullingTime="275.658169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:48:03.717616053 +0000 UTC m=+860.015259222" watchObservedRunningTime="2026-07-08 17:48:03.718081601 +0000 UTC m=+860.015724787" Jul 08 17:48:06.961426 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:06.961391 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:48:06.961866 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:06.961485 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:48:06.973363 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:06.973340 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:48:07.730114 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:07.730081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-bjbg9" Jul 08 17:48:13.178110 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:13.178072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:13.178537 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:13.178149 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:13.190620 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:13.190597 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:13.756761 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:13.756732 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz" Jul 08 17:48:17.589630 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:17.589596 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:28.861737 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.861707 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz"] Jul 08 17:48:28.867622 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.867599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.869847 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.869824 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 17:48:28.872922 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.872901 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz"] Jul 08 17:48:28.888642 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.888610 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:28.949870 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.949840 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5180e1-7290-4c71-bc42-665d2c07b86a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.949990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.949899 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.949990 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.949922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.950110 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.950036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.950110 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.950095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpnkd\" (UniqueName: \"kubernetes.io/projected/da5180e1-7290-4c71-bc42-665d2c07b86a-kube-api-access-kpnkd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:28.950210 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:28.950192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051722 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5180e1-7290-4c71-bc42-665d2c07b86a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.051915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.051856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpnkd\" (UniqueName: \"kubernetes.io/projected/da5180e1-7290-4c71-bc42-665d2c07b86a-kube-api-access-kpnkd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.052201 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.052178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.052253 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.052200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.052288 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.052244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.054216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.054186 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da5180e1-7290-4c71-bc42-665d2c07b86a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.054480 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.054462 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da5180e1-7290-4c71-bc42-665d2c07b86a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.059712 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.059692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpnkd\" (UniqueName: \"kubernetes.io/projected/da5180e1-7290-4c71-bc42-665d2c07b86a-kube-api-access-kpnkd\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz\" (UID: \"da5180e1-7290-4c71-bc42-665d2c07b86a\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.201915 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.201825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:29.337033 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.336883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz"] Jul 08 17:48:29.339938 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:48:29.339909 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda5180e1_7290_4c71_bc42_665d2c07b86a.slice/crio-8e805df345a6635f99bb2112789019aa89d2695cf61d3d8b751b8cc48963e2d9 WatchSource:0}: Error finding container 8e805df345a6635f99bb2112789019aa89d2695cf61d3d8b751b8cc48963e2d9: Status 404 returned error can't find the container with id 8e805df345a6635f99bb2112789019aa89d2695cf61d3d8b751b8cc48963e2d9 Jul 08 17:48:29.818002 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.817965 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" event={"ID":"da5180e1-7290-4c71-bc42-665d2c07b86a","Type":"ContainerStarted","Data":"3ab6d50434e0340f0bc6bba0aeecd98726f60686807185c51b56c8ebc3cf59c6"} Jul 08 17:48:29.818163 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.818011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" event={"ID":"da5180e1-7290-4c71-bc42-665d2c07b86a","Type":"ContainerStarted","Data":"8e805df345a6635f99bb2112789019aa89d2695cf61d3d8b751b8cc48963e2d9"} Jul 08 17:48:29.836873 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:29.836819 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" podStartSLOduration=1.534225874 podStartE2EDuration="1.836803326s" totalImagesPullingTime="302.577452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:48:29.835203656 +0000 UTC m=+886.132846823" watchObservedRunningTime="2026-07-08 17:48:29.836803326 +0000 UTC m=+886.134446492" Jul 08 17:48:39.202333 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:39.202233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:39.202790 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:39.202497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:39.214809 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:39.214784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:48:39.875136 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:48:39.875096 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz" Jul 08 17:49:15.334713 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.334683 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-786cbd6df6-t9p8d"] Jul 08 17:49:15.337977 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.337962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.346741 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.346713 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-786cbd6df6-t9p8d"] Jul 08 17:49:15.474703 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.474672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2326706f-69fc-414a-bf1f-b85b1e2f6baf-tls-cert\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.474852 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.474716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68cmn\" (UniqueName: \"kubernetes.io/projected/2326706f-69fc-414a-bf1f-b85b1e2f6baf-kube-api-access-68cmn\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.576213 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.576184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2326706f-69fc-414a-bf1f-b85b1e2f6baf-tls-cert\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.576213 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.576215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68cmn\" (UniqueName: \"kubernetes.io/projected/2326706f-69fc-414a-bf1f-b85b1e2f6baf-kube-api-access-68cmn\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.578744 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.578726 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2326706f-69fc-414a-bf1f-b85b1e2f6baf-tls-cert\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.584994 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.584931 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68cmn\" (UniqueName: \"kubernetes.io/projected/2326706f-69fc-414a-bf1f-b85b1e2f6baf-kube-api-access-68cmn\") pod \"authorino-786cbd6df6-t9p8d\" (UID: \"2326706f-69fc-414a-bf1f-b85b1e2f6baf\") " pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.648361 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.648325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-786cbd6df6-t9p8d" Jul 08 17:49:15.788872 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.788848 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-786cbd6df6-t9p8d"] Jul 08 17:49:15.790824 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:49:15.790798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2326706f_69fc_414a_bf1f_b85b1e2f6baf.slice/crio-71433e797ef9dd0fb19814bcac72c3483eead61baf73daf9e02eb5d123e1b580 WatchSource:0}: Error finding container 71433e797ef9dd0fb19814bcac72c3483eead61baf73daf9e02eb5d123e1b580: Status 404 returned error can't find the container with id 71433e797ef9dd0fb19814bcac72c3483eead61baf73daf9e02eb5d123e1b580 Jul 08 17:49:15.838887 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.838804 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:15.937475 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:15.937435 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:16.030877 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:16.030848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-786cbd6df6-t9p8d" event={"ID":"2326706f-69fc-414a-bf1f-b85b1e2f6baf","Type":"ContainerStarted","Data":"71433e797ef9dd0fb19814bcac72c3483eead61baf73daf9e02eb5d123e1b580"} Jul 08 17:49:17.038709 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.038672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-786cbd6df6-t9p8d" event={"ID":"2326706f-69fc-414a-bf1f-b85b1e2f6baf","Type":"ContainerStarted","Data":"0829ae621881c7eb8efba25fb876bf3e6ad3a450d4c9d596b058d94f1ad8da6c"} Jul 08 17:49:17.179221 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.179186 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:17.205881 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.205797 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-786cbd6df6-t9p8d" podStartSLOduration=1.830330695 podStartE2EDuration="2.205775898s" totalImagesPullingTime="375.445203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:49:17.105471767 +0000 UTC m=+933.403114931" watchObservedRunningTime="2026-07-08 17:49:17.205775898 +0000 UTC m=+933.503419070" Jul 08 17:49:17.206721 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.206701 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:49:17.206877 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.206859 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" podUID="4b7bc925-89c5-487c-8409-7646cffcaa7a" containerName="authorino" containerID="cri-o://e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff" gracePeriod=30 Jul 08 17:49:17.492475 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.492450 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:49:17.594033 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.593945 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22b7\" (UniqueName: \"kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7\") pod \"4b7bc925-89c5-487c-8409-7646cffcaa7a\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " Jul 08 17:49:17.594198 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.594033 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert\") pod \"4b7bc925-89c5-487c-8409-7646cffcaa7a\" (UID: \"4b7bc925-89c5-487c-8409-7646cffcaa7a\") " Jul 08 17:49:17.596850 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.596823 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7" (OuterVolumeSpecName: "kube-api-access-g22b7") pod "4b7bc925-89c5-487c-8409-7646cffcaa7a" (UID: "4b7bc925-89c5-487c-8409-7646cffcaa7a"). InnerVolumeSpecName "kube-api-access-g22b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:49:17.605404 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.605381 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4b7bc925-89c5-487c-8409-7646cffcaa7a" (UID: "4b7bc925-89c5-487c-8409-7646cffcaa7a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:49:17.695373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.695339 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b7bc925-89c5-487c-8409-7646cffcaa7a-tls-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:49:17.695373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.695369 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22b7\" (UniqueName: \"kubernetes.io/projected/4b7bc925-89c5-487c-8409-7646cffcaa7a-kube-api-access-g22b7\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:49:17.932047 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:17.932013 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:18.047913 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.047864 2578 generic.go:356] "Generic (PLEG): container finished" podID="4b7bc925-89c5-487c-8409-7646cffcaa7a" containerID="e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff" exitCode=0 Jul 08 17:49:18.047913 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.047913 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" Jul 08 17:49:18.048412 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.047955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" event={"ID":"4b7bc925-89c5-487c-8409-7646cffcaa7a","Type":"ContainerDied","Data":"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff"} Jul 08 17:49:18.048412 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.048008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c9f44d57-pqgpf" event={"ID":"4b7bc925-89c5-487c-8409-7646cffcaa7a","Type":"ContainerDied","Data":"5383252c87655315f0b71e606a1a6b78f640d80df386883dd643d3cc9d84a81e"} Jul 08 17:49:18.048412 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.048030 2578 scope.go:117] "RemoveContainer" containerID="e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff" Jul 08 17:49:18.068694 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.068656 2578 scope.go:117] "RemoveContainer" containerID="e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff" Jul 08 17:49:18.069808 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:49:18.069772 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff\": container with ID starting with e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff not found: ID does not exist" containerID="e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff" Jul 08 17:49:18.069917 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.069820 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff"} err="failed to get container status \"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff\": rpc error: code = NotFound desc = could not find container \"e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff\": container with ID starting with e73b2659945a6808a485322a98815fa323c4a23fcb52eef5f6066018941442ff not found: ID does not exist" Jul 08 17:49:18.095217 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.095192 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:49:18.103418 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.103391 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75c9f44d57-pqgpf"] Jul 08 17:49:18.309468 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:18.309390 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7bc925-89c5-487c-8409-7646cffcaa7a" path="/var/lib/kubelet/pods/4b7bc925-89c5-487c-8409-7646cffcaa7a/volumes" Jul 08 17:49:48.258268 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:48.258222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:48.359530 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:49:48.359489 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:23.840231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:23.840136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:28.523344 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.523308 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:28.626586 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.626557 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:50:28.626835 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.626804 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-67df7499f4-z2rtm" podUID="1c118e60-d957-4081-837c-37f047041690" containerName="manager" containerID="cri-o://3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18" gracePeriod=30 Jul 08 17:50:28.879133 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.879098 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:50:28.953638 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.953613 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqpcf\" (UniqueName: \"kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf\") pod \"1c118e60-d957-4081-837c-37f047041690\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " Jul 08 17:50:28.953827 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.953700 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert\") pod \"1c118e60-d957-4081-837c-37f047041690\" (UID: \"1c118e60-d957-4081-837c-37f047041690\") " Jul 08 17:50:28.955708 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.955676 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf" (OuterVolumeSpecName: "kube-api-access-fqpcf") pod "1c118e60-d957-4081-837c-37f047041690" (UID: "1c118e60-d957-4081-837c-37f047041690"). InnerVolumeSpecName "kube-api-access-fqpcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:50:28.955708 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:28.955676 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1c118e60-d957-4081-837c-37f047041690" (UID: "1c118e60-d957-4081-837c-37f047041690"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:50:29.054408 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.054353 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpcf\" (UniqueName: \"kubernetes.io/projected/1c118e60-d957-4081-837c-37f047041690-kube-api-access-fqpcf\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:50:29.054408 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.054377 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c118e60-d957-4081-837c-37f047041690-webhook-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 17:50:29.368229 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.368197 2578 generic.go:356] "Generic (PLEG): container finished" podID="1c118e60-d957-4081-837c-37f047041690" containerID="3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18" exitCode=0 Jul 08 17:50:29.368351 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.368240 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-z2rtm" Jul 08 17:50:29.368351 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.368268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-z2rtm" event={"ID":"1c118e60-d957-4081-837c-37f047041690","Type":"ContainerDied","Data":"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18"} Jul 08 17:50:29.368351 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.368310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-z2rtm" event={"ID":"1c118e60-d957-4081-837c-37f047041690","Type":"ContainerDied","Data":"e04f87fa148ff3f5b6efd158764766a8efd6c66ad7fbbd3a59c2785dd9e2e4a0"} Jul 08 17:50:29.368351 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.368332 2578 scope.go:117] "RemoveContainer" containerID="3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18" Jul 08 17:50:29.377146 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.377130 2578 scope.go:117] "RemoveContainer" containerID="3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18" Jul 08 17:50:29.377405 ip-10-0-134-201 kubenswrapper[2578]: E0708 17:50:29.377387 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18\": container with ID starting with 3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18 not found: ID does not exist" containerID="3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18" Jul 08 17:50:29.377457 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.377416 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18"} err="failed to get container status \"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18\": rpc error: code = NotFound desc = could not find container \"3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18\": container with ID starting with 3b3fac632d0e8c8381fd8913ef5c373cb26dd87ff12ff86eb22d870e82581e18 not found: ID does not exist" Jul 08 17:50:29.391819 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.391793 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:50:29.395800 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:29.395780 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67df7499f4-z2rtm"] Jul 08 17:50:30.113961 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.113927 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:30.195854 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.195824 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67df7499f4-gmf7j"] Jul 08 17:50:30.196163 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196150 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c118e60-d957-4081-837c-37f047041690" containerName="manager" Jul 08 17:50:30.196216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196165 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c118e60-d957-4081-837c-37f047041690" containerName="manager" Jul 08 17:50:30.196216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196186 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7bc925-89c5-487c-8409-7646cffcaa7a" containerName="authorino" Jul 08 17:50:30.196216 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196192 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7bc925-89c5-487c-8409-7646cffcaa7a" containerName="authorino" Jul 08 17:50:30.196312 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196261 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c118e60-d957-4081-837c-37f047041690" containerName="manager" Jul 08 17:50:30.196312 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.196270 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7bc925-89c5-487c-8409-7646cffcaa7a" containerName="authorino" Jul 08 17:50:30.200474 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.200460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.203755 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.203733 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6xqmn\"" Jul 08 17:50:30.204144 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.204128 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 17:50:30.204206 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.204161 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 17:50:30.213138 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.213117 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67df7499f4-gmf7j"] Jul 08 17:50:30.264067 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.264035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld7w4\" (UniqueName: \"kubernetes.io/projected/a646702a-4ef7-4f54-aea7-4309b117e277-kube-api-access-ld7w4\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.264231 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.264071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a646702a-4ef7-4f54-aea7-4309b117e277-webhook-cert\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.310034 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.310000 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c118e60-d957-4081-837c-37f047041690" path="/var/lib/kubelet/pods/1c118e60-d957-4081-837c-37f047041690/volumes" Jul 08 17:50:30.365061 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.364980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld7w4\" (UniqueName: \"kubernetes.io/projected/a646702a-4ef7-4f54-aea7-4309b117e277-kube-api-access-ld7w4\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.365061 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.365019 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a646702a-4ef7-4f54-aea7-4309b117e277-webhook-cert\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.367609 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.367586 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a646702a-4ef7-4f54-aea7-4309b117e277-webhook-cert\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.374544 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.374501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld7w4\" (UniqueName: \"kubernetes.io/projected/a646702a-4ef7-4f54-aea7-4309b117e277-kube-api-access-ld7w4\") pod \"maas-controller-67df7499f4-gmf7j\" (UID: \"a646702a-4ef7-4f54-aea7-4309b117e277\") " pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.510672 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.510642 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:30.640799 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.640745 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67df7499f4-gmf7j"] Jul 08 17:50:30.643017 ip-10-0-134-201 kubenswrapper[2578]: W0708 17:50:30.642990 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda646702a_4ef7_4f54_aea7_4309b117e277.slice/crio-7281719bab4cb0b2fd6d0618a06990462544c29fb721132631d81294d9674b40 WatchSource:0}: Error finding container 7281719bab4cb0b2fd6d0618a06990462544c29fb721132631d81294d9674b40: Status 404 returned error can't find the container with id 7281719bab4cb0b2fd6d0618a06990462544c29fb721132631d81294d9674b40 Jul 08 17:50:30.644373 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:30.644357 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:50:31.379717 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:31.379676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-gmf7j" event={"ID":"a646702a-4ef7-4f54-aea7-4309b117e277","Type":"ContainerStarted","Data":"aab96b3dffba00c2a724696120223954c0e02d3b1c663248e16c7cfc10242213"} Jul 08 17:50:31.380131 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:31.379857 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:50:31.380131 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:31.379879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67df7499f4-gmf7j" event={"ID":"a646702a-4ef7-4f54-aea7-4309b117e277","Type":"ContainerStarted","Data":"7281719bab4cb0b2fd6d0618a06990462544c29fb721132631d81294d9674b40"} Jul 08 17:50:31.398454 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:31.398369 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67df7499f4-gmf7j" podStartSLOduration=0.90599219 podStartE2EDuration="1.398355308s" totalImagesPullingTime="492.363118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:50:31.396875393 +0000 UTC m=+1007.694518559" watchObservedRunningTime="2026-07-08 17:50:31.398355308 +0000 UTC m=+1007.695998493" Jul 08 17:50:42.389953 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:50:42.389920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67df7499f4-gmf7j" Jul 08 17:51:03.463276 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:03.463245 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:03.561739 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:03.561700 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:03.685080 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:03.685048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:09.530224 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:09.530187 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:09.635473 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:09.635439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:09.730726 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:09.730696 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:09.875347 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:09.875294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:15.735477 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:15.735436 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:15.836357 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:15.836322 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:33.801726 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:33.801639 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:33.903120 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:51:33.903088 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:52:35.250555 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:52:35.250501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:52:58.222452 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:52:58.222413 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:52:58.876921 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:52:58.876887 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:52:59.055197 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:52:59.055161 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:36.877477 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:53:36.877388 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:36.940080 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:53:36.940050 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:55:01.411936 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:55:01.411901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:59:12.103188 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:59:12.103086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:59:23.706942 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:59:23.706900 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 17:59:33.074462 ip-10-0-134-201 kubenswrapper[2578]: I0708 17:59:33.074416 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:18.698579 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:18.698483 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:18.980672 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:18.980586 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:21.015644 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:21.015613 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:32.854714 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:32.854681 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:41.121377 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:41.121343 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:46.327917 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:46.327881 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:49.343441 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:49.343395 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:49.458530 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:49.458482 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:49.928846 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:49.928807 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:55.857092 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:55.857055 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:55.936633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:55.936599 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:56.055722 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.055692 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:02:56.059170 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.059098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.061623 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.061603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b00c7927-openshift-default-dockercfg-kl2pt\"" Jul 08 18:02:56.075731 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.075705 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:02:56.155717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.155717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.155717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.155717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf4z\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.156060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.156060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.156060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.156060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.156060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.155981 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.192089 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.192060 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:56.256919 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.256885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.256931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.256958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf4z\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257292 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257292 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257292 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257444 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257493 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257569 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257713 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257689 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257867 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.257951 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.257803 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:56.258072 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.258055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.259691 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.259664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.260039 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.260018 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.265795 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.265775 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.266177 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.266155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf4z\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z\") pod \"e2e-ait-b00c7927-openshift-default-799c695987-c2fdq\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.369370 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.369338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:56.510017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.509977 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:02:56.513994 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:02:56.513962 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304495d0_f5cf_4a21_a6fa_995cc6bec563.slice/crio-247e70119f4cbaee3cbb46dab71722c91687cdc19750a8eed87f65dc141624bb WatchSource:0}: Error finding container 247e70119f4cbaee3cbb46dab71722c91687cdc19750a8eed87f65dc141624bb: Status 404 returned error can't find the container with id 247e70119f4cbaee3cbb46dab71722c91687cdc19750a8eed87f65dc141624bb Jul 08 18:02:56.516133 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.516100 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:02:56.516206 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.516172 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:02:56.516206 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.516202 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:02:56.849270 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.849166 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" event={"ID":"304495d0-f5cf-4a21-a6fa-995cc6bec563","Type":"ContainerStarted","Data":"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde"} Jul 08 18:02:56.849270 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.849261 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" event={"ID":"304495d0-f5cf-4a21-a6fa-995cc6bec563","Type":"ContainerStarted","Data":"247e70119f4cbaee3cbb46dab71722c91687cdc19750a8eed87f65dc141624bb"} Jul 08 18:02:56.869145 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:56.869098 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" podStartSLOduration=0.869082226 podStartE2EDuration="869.082226ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:02:56.86740388 +0000 UTC m=+1753.165047049" watchObservedRunningTime="2026-07-08 18:02:56.869082226 +0000 UTC m=+1753.166725393" Jul 08 18:02:57.370427 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.370392 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:57.371984 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.371957 2578 patch_prober.go:28] interesting pod/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 08 18:02:57.372122 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.372016 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 08 18:02:57.502510 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.502478 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:02:57.506287 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.506269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.508426 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.508393 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:57.509574 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.509552 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b00c7927-serving-cert\"" Jul 08 18:02:57.520428 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.520407 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:02:57.596412 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.596367 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:57.671758 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.671668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbrj\" (UniqueName: \"kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.671929 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.671755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.671929 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.671842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.772778 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.772742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.772954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.772812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.772954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.772846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbrj\" (UniqueName: \"kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.773039 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:02:57.772973 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-serving-cert: secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:02:57.773039 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:02:57.773034 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls podName:f6894f68-cfcc-4a7e-9529-06545fef64c3 nodeName:}" failed. No retries permitted until 2026-07-08 18:02:58.273018874 +0000 UTC m=+1754.570662019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls") pod "maas-api-e2e-ait-b00c7927-7df5466699-597xk" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3") : secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:02:57.773560 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.773538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:57.786366 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:57.786344 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbrj\" (UniqueName: \"kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:58.047836 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.047744 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:58.150037 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.149998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:58.278589 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.278546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:58.278772 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:02:58.278697 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-serving-cert: secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:02:58.278772 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:02:58.278763 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls podName:f6894f68-cfcc-4a7e-9529-06545fef64c3 nodeName:}" failed. No retries permitted until 2026-07-08 18:02:59.278746873 +0000 UTC m=+1755.576390021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls") pod "maas-api-e2e-ait-b00c7927-7df5466699-597xk" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3") : secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:02:58.374394 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.374361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:58.523151 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.523093 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:02:58.866013 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.865977 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:58.867051 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:58.867033 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:02:59.288426 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.288329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:59.290861 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.290841 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"maas-api-e2e-ait-b00c7927-7df5466699-597xk\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:59.319025 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.318979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:02:59.456480 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.456447 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:02:59.459681 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:02:59.459651 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6894f68_cfcc_4a7e_9529_06545fef64c3.slice/crio-81ae3cf665e945af481213c50781146bcb1fb6aab823c8e4aff6b602de38ec40 WatchSource:0}: Error finding container 81ae3cf665e945af481213c50781146bcb1fb6aab823c8e4aff6b602de38ec40: Status 404 returned error can't find the container with id 81ae3cf665e945af481213c50781146bcb1fb6aab823c8e4aff6b602de38ec40 Jul 08 18:02:59.461172 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.461156 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:02:59.871591 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:02:59.871549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" event={"ID":"f6894f68-cfcc-4a7e-9529-06545fef64c3","Type":"ContainerStarted","Data":"81ae3cf665e945af481213c50781146bcb1fb6aab823c8e4aff6b602de38ec40"} Jul 08 18:03:00.608549 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:00.608498 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:01.881809 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:01.881767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" event={"ID":"f6894f68-cfcc-4a7e-9529-06545fef64c3","Type":"ContainerStarted","Data":"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5"} Jul 08 18:03:01.882292 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:01.881939 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:03:01.904889 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:01.904813 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" podStartSLOduration=3.127863445 podStartE2EDuration="4.904789482s" totalImagesPullingTime="1.776926037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:03:01.901555083 +0000 UTC m=+1758.199198250" watchObservedRunningTime="2026-07-08 18:03:01.904789482 +0000 UTC m=+1758.202432650" Jul 08 18:03:02.401327 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.401264 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:02.586112 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.586067 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:03:02.586342 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.586310 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerName="istio-proxy" containerID="cri-o://d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde" gracePeriod=30 Jul 08 18:03:02.612674 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.612624 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:02.712162 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.712060 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:02.801854 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.801069 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:03:02.823346 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:02.823315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:02.925776 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:02.925734 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-serving-cert: secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:03:02.926154 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:02.925845 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls podName:f6894f68-cfcc-4a7e-9529-06545fef64c3 nodeName:}" failed. No retries permitted until 2026-07-08 18:03:03.425819786 +0000 UTC m=+1759.723462957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls") pod "maas-api-e2e-ait-b00c7927-7df5466699-597xk" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3") : secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:03:03.429893 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:03.429855 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-serving-cert: secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:03:03.430083 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:03.429942 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls podName:f6894f68-cfcc-4a7e-9529-06545fef64c3 nodeName:}" failed. No retries permitted until 2026-07-08 18:03:04.429928198 +0000 UTC m=+1760.727571343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls") pod "maas-api-e2e-ait-b00c7927-7df5466699-597xk" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3") : secret "maas-api-e2e-ait-b00c7927-serving-cert" not found Jul 08 18:03:03.907313 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:03.907259 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" podUID="f6894f68-cfcc-4a7e-9529-06545fef64c3" containerName="maas-api" containerID="cri-o://395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5" gracePeriod=30 Jul 08 18:03:04.167148 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.167072 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:03:04.237020 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.236984 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") pod \"f6894f68-cfcc-4a7e-9529-06545fef64c3\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " Jul 08 18:03:04.237193 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.237058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmbrj\" (UniqueName: \"kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj\") pod \"f6894f68-cfcc-4a7e-9529-06545fef64c3\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " Jul 08 18:03:04.237193 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.237093 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca\") pod \"f6894f68-cfcc-4a7e-9529-06545fef64c3\" (UID: \"f6894f68-cfcc-4a7e-9529-06545fef64c3\") " Jul 08 18:03:04.237534 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.237487 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6894f68-cfcc-4a7e-9529-06545fef64c3" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:04.239413 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.239388 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6894f68-cfcc-4a7e-9529-06545fef64c3" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:03:04.239583 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.239562 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj" (OuterVolumeSpecName: "kube-api-access-mmbrj") pod "f6894f68-cfcc-4a7e-9529-06545fef64c3" (UID: "f6894f68-cfcc-4a7e-9529-06545fef64c3"). InnerVolumeSpecName "kube-api-access-mmbrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:04.338690 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.338654 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbrj\" (UniqueName: \"kubernetes.io/projected/f6894f68-cfcc-4a7e-9529-06545fef64c3-kube-api-access-mmbrj\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:04.338690 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.338684 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6894f68-cfcc-4a7e-9529-06545fef64c3-openshift-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:04.338690 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.338694 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6894f68-cfcc-4a7e-9529-06545fef64c3-maas-api-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:04.912682 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.912645 2578 generic.go:356] "Generic (PLEG): container finished" podID="f6894f68-cfcc-4a7e-9529-06545fef64c3" containerID="395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5" exitCode=0 Jul 08 18:03:04.912876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.912694 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" Jul 08 18:03:04.912876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.912714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" event={"ID":"f6894f68-cfcc-4a7e-9529-06545fef64c3","Type":"ContainerDied","Data":"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5"} Jul 08 18:03:04.912876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.912758 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk" event={"ID":"f6894f68-cfcc-4a7e-9529-06545fef64c3","Type":"ContainerDied","Data":"81ae3cf665e945af481213c50781146bcb1fb6aab823c8e4aff6b602de38ec40"} Jul 08 18:03:04.912876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.912782 2578 scope.go:117] "RemoveContainer" containerID="395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5" Jul 08 18:03:04.921574 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.921549 2578 scope.go:117] "RemoveContainer" containerID="395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5" Jul 08 18:03:04.921886 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:04.921866 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5\": container with ID starting with 395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5 not found: ID does not exist" containerID="395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5" Jul 08 18:03:04.921928 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.921896 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5"} err="failed to get container status \"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5\": rpc error: code = NotFound desc = could not find container \"395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5\": container with ID starting with 395637a6606ae1122a55c4412b5201efcf57969c94843fad70d2b8d1cc9b60d5 not found: ID does not exist" Jul 08 18:03:04.928750 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.928713 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:03:04.932860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:04.932833 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b00c7927-7df5466699-597xk"] Jul 08 18:03:06.078725 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:06.078682 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:06.310618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:06.310535 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6894f68-cfcc-4a7e-9529-06545fef64c3" path="/var/lib/kubelet/pods/f6894f68-cfcc-4a7e-9529-06545fef64c3/volumes" Jul 08 18:03:08.535968 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.535908 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:03:08.584152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584125 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584301 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584156 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584301 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584210 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584301 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584280 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584464 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584321 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584464 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584349 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584464 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584412 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584637 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584504 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:08.584637 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584606 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:08.584819 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584800 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:08.584879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584847 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:08.584919 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584893 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnf4z\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.584953 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.584939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy\") pod \"304495d0-f5cf-4a21-a6fa-995cc6bec563\" (UID: \"304495d0-f5cf-4a21-a6fa-995cc6bec563\") " Jul 08 18:03:08.585037 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585019 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data" (OuterVolumeSpecName: "istio-data") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:08.585302 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585285 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.585439 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585304 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.585439 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585313 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.585439 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585322 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304495d0-f5cf-4a21-a6fa-995cc6bec563-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.585439 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.585331 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.587284 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.587248 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:03:08.587284 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.587265 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token" (OuterVolumeSpecName: "istio-token") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:08.587430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.587360 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z" (OuterVolumeSpecName: "kube-api-access-lnf4z") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "kube-api-access-lnf4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:08.587509 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.587487 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "304495d0-f5cf-4a21-a6fa-995cc6bec563" (UID: "304495d0-f5cf-4a21-a6fa-995cc6bec563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:08.686334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.686256 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.686334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.686282 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.686334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.686291 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnf4z\" (UniqueName: \"kubernetes.io/projected/304495d0-f5cf-4a21-a6fa-995cc6bec563-kube-api-access-lnf4z\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.686334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.686301 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304495d0-f5cf-4a21-a6fa-995cc6bec563-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:08.796345 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.796315 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:08.904688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.904654 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:08.936703 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.936626 2578 generic.go:356] "Generic (PLEG): container finished" podID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerID="d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde" exitCode=0 Jul 08 18:03:08.936703 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.936672 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" Jul 08 18:03:08.936703 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.936678 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" event={"ID":"304495d0-f5cf-4a21-a6fa-995cc6bec563","Type":"ContainerDied","Data":"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde"} Jul 08 18:03:08.936936 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.936720 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq" event={"ID":"304495d0-f5cf-4a21-a6fa-995cc6bec563","Type":"ContainerDied","Data":"247e70119f4cbaee3cbb46dab71722c91687cdc19750a8eed87f65dc141624bb"} Jul 08 18:03:08.936936 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.936741 2578 scope.go:117] "RemoveContainer" containerID="d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde" Jul 08 18:03:08.945827 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.945809 2578 scope.go:117] "RemoveContainer" containerID="d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde" Jul 08 18:03:08.946093 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:08.946076 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde\": container with ID starting with d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde not found: ID does not exist" containerID="d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde" Jul 08 18:03:08.946139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.946100 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde"} err="failed to get container status \"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde\": rpc error: code = NotFound desc = could not find container \"d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde\": container with ID starting with d6bdd1e537949f0816129380da94d416ca1c7e2715b71db94699885329af0bde not found: ID does not exist" Jul 08 18:03:08.963938 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.963910 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:03:08.975028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:08.975003 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b00c7927-openshift-default-799c695987-c2fdq"] Jul 08 18:03:09.000121 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000088 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:09.000461 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000447 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6894f68-cfcc-4a7e-9529-06545fef64c3" containerName="maas-api" Jul 08 18:03:09.000593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000463 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6894f68-cfcc-4a7e-9529-06545fef64c3" containerName="maas-api" Jul 08 18:03:09.000593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000474 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerName="istio-proxy" Jul 08 18:03:09.000593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000479 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerName="istio-proxy" Jul 08 18:03:09.000593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000576 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6894f68-cfcc-4a7e-9529-06545fef64c3" containerName="maas-api" Jul 08 18:03:09.000593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.000589 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" containerName="istio-proxy" Jul 08 18:03:09.003735 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.003718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.006379 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.006360 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-141dfad7-openshift-default-dockercfg-6tmjp\"" Jul 08 18:03:09.022116 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.022094 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:09.051769 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.051736 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:09.090183 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090183 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4c4d\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090362 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.090579 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.090378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191271 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191271 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191271 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191809 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191871 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191871 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191982 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4c4d\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.191982 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.191962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.192092 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.192009 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.192185 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.192162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.193881 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.193859 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.194248 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.194231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.203359 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.203336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.204664 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.204637 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:09.210543 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.210508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4c4d\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d\") pod \"e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.314624 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.314590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:09.474724 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.474287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:09.477271 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:03:09.477243 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3adff20_0f11_400b_b805_180f82153d5c.slice/crio-d82353d9912b1bb6c072e63effcd87ffd39b4b5df3add85a0786b3e09a129aad WatchSource:0}: Error finding container d82353d9912b1bb6c072e63effcd87ffd39b4b5df3add85a0786b3e09a129aad: Status 404 returned error can't find the container with id d82353d9912b1bb6c072e63effcd87ffd39b4b5df3add85a0786b3e09a129aad Jul 08 18:03:09.479495 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.479457 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:09.479584 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.479564 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:09.479630 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.479614 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:09.943603 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.943567 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" event={"ID":"a3adff20-0f11-400b-b805-180f82153d5c","Type":"ContainerStarted","Data":"617c045fc81453b5c598cf5e30eb6d45a43295d90a46a529100cb7ffe27639ec"} Jul 08 18:03:09.943977 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.943619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" event={"ID":"a3adff20-0f11-400b-b805-180f82153d5c","Type":"ContainerStarted","Data":"d82353d9912b1bb6c072e63effcd87ffd39b4b5df3add85a0786b3e09a129aad"} Jul 08 18:03:09.968085 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:09.968037 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" podStartSLOduration=1.968021129 podStartE2EDuration="1.968021129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:03:09.965215684 +0000 UTC m=+1766.262858851" watchObservedRunningTime="2026-07-08 18:03:09.968021129 +0000 UTC m=+1766.265664296" Jul 08 18:03:10.245749 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.245645 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:10.310286 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.310251 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="304495d0-f5cf-4a21-a6fa-995cc6bec563" path="/var/lib/kubelet/pods/304495d0-f5cf-4a21-a6fa-995cc6bec563/volumes" Jul 08 18:03:10.315370 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.315353 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:10.320363 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.320342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:10.363256 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.363222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:10.371795 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.371770 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:10.374318 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.374304 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.382986 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.382966 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-141dfad7-serving-cert\"" Jul 08 18:03:10.402914 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.402886 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:10.403336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.403308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.403453 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.403437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmhq\" (UniqueName: \"kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.403535 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.403481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.504876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.504789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.505020 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.504876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmhq\" (UniqueName: \"kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.505020 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.504899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.505020 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:10.504955 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-serving-cert: secret "maas-api-e2e-ait-141dfad7-serving-cert" not found Jul 08 18:03:10.505126 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:10.505041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls podName:5115f6e1-5768-4bb0-aaf4-1e1931a8025f nodeName:}" failed. No retries permitted until 2026-07-08 18:03:11.005021338 +0000 UTC m=+1767.302664497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls") pod "maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" (UID: "5115f6e1-5768-4bb0-aaf4-1e1931a8025f") : secret "maas-api-e2e-ait-141dfad7-serving-cert" not found Jul 08 18:03:10.505563 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.505539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.506190 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.506151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:10.517162 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.517136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmhq\" (UniqueName: \"kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:10.651647 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.651616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:10.948083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.948050 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:10.948939 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:10.948920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:11.010664 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.010634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:11.013413 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.013384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") pod \"maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:11.266055 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.265971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:11.284610 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.284553 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:11.386361 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.386319 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:11.441057 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.441018 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:11.442831 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:03:11.442787 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115f6e1_5768_4bb0_aaf4_1e1931a8025f.slice/crio-c1dcf9c10544a26efafbf3d0cc617d42fc977a7062c667a4dcc88bef6bf359f3 WatchSource:0}: Error finding container c1dcf9c10544a26efafbf3d0cc617d42fc977a7062c667a4dcc88bef6bf359f3: Status 404 returned error can't find the container with id c1dcf9c10544a26efafbf3d0cc617d42fc977a7062c667a4dcc88bef6bf359f3 Jul 08 18:03:11.953505 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:11.953471 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" event={"ID":"5115f6e1-5768-4bb0-aaf4-1e1931a8025f","Type":"ContainerStarted","Data":"c1dcf9c10544a26efafbf3d0cc617d42fc977a7062c667a4dcc88bef6bf359f3"} Jul 08 18:03:12.960088 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:12.960057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" event={"ID":"5115f6e1-5768-4bb0-aaf4-1e1931a8025f","Type":"ContainerStarted","Data":"6e9e5b283090a935a8ff8ab31595f3fb98e1811eb80e75bd5025312f9e43af2d"} Jul 08 18:03:12.960469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:12.960157 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:12.979609 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:12.979564 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" podStartSLOduration=2.438963618 podStartE2EDuration="2.979550093s" totalImagesPullingTime="540.586475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:03:12.977885601 +0000 UTC m=+1769.275528767" watchObservedRunningTime="2026-07-08 18:03:12.979550093 +0000 UTC m=+1769.277193260" Jul 08 18:03:13.255342 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:13.255312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:15.373880 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:15.373839 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:15.784336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:15.783482 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:15.784336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:15.783747 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" podUID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" containerName="maas-api" containerID="cri-o://6e9e5b283090a935a8ff8ab31595f3fb98e1811eb80e75bd5025312f9e43af2d" gracePeriod=30 Jul 08 18:03:15.809209 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:15.809070 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:16.009697 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:15.994372 2578 generic.go:356] "Generic (PLEG): container finished" podID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" containerID="6e9e5b283090a935a8ff8ab31595f3fb98e1811eb80e75bd5025312f9e43af2d" exitCode=0 Jul 08 18:03:16.017958 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.002233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" event={"ID":"5115f6e1-5768-4bb0-aaf4-1e1931a8025f","Type":"ContainerDied","Data":"6e9e5b283090a935a8ff8ab31595f3fb98e1811eb80e75bd5025312f9e43af2d"} Jul 08 18:03:16.070247 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.070221 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:16.163335 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.163285 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca\") pod \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " Jul 08 18:03:16.163553 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.163354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzmhq\" (UniqueName: \"kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq\") pod \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " Jul 08 18:03:16.163553 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.163385 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") pod \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\" (UID: \"5115f6e1-5768-4bb0-aaf4-1e1931a8025f\") " Jul 08 18:03:16.163698 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.163673 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5115f6e1-5768-4bb0-aaf4-1e1931a8025f" (UID: "5115f6e1-5768-4bb0-aaf4-1e1931a8025f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:16.165787 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.165749 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5115f6e1-5768-4bb0-aaf4-1e1931a8025f" (UID: "5115f6e1-5768-4bb0-aaf4-1e1931a8025f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:03:16.165907 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.165826 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq" (OuterVolumeSpecName: "kube-api-access-fzmhq") pod "5115f6e1-5768-4bb0-aaf4-1e1931a8025f" (UID: "5115f6e1-5768-4bb0-aaf4-1e1931a8025f"). InnerVolumeSpecName "kube-api-access-fzmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:16.264710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.264671 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmhq\" (UniqueName: \"kubernetes.io/projected/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-kube-api-access-fzmhq\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:16.264710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.264702 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-maas-api-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:16.264710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:16.264711 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5115f6e1-5768-4bb0-aaf4-1e1931a8025f-openshift-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:17.022167 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:17.022132 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" event={"ID":"5115f6e1-5768-4bb0-aaf4-1e1931a8025f","Type":"ContainerDied","Data":"c1dcf9c10544a26efafbf3d0cc617d42fc977a7062c667a4dcc88bef6bf359f3"} Jul 08 18:03:17.022167 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:17.022150 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b" Jul 08 18:03:17.022720 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:17.022200 2578 scope.go:117] "RemoveContainer" containerID="6e9e5b283090a935a8ff8ab31595f3fb98e1811eb80e75bd5025312f9e43af2d" Jul 08 18:03:17.046191 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:17.046168 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:17.049944 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:17.049923 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-141dfad7-7dbd59dbd7-5w84b"] Jul 08 18:03:18.310811 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:18.310774 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" path="/var/lib/kubelet/pods/5115f6e1-5768-4bb0-aaf4-1e1931a8025f/volumes" Jul 08 18:03:21.459061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:21.459029 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:21.459566 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:21.459236 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" podUID="a3adff20-0f11-400b-b805-180f82153d5c" containerName="istio-proxy" containerID="cri-o://617c045fc81453b5c598cf5e30eb6d45a43295d90a46a529100cb7ffe27639ec" gracePeriod=30 Jul 08 18:03:21.485192 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:21.485156 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:21.589326 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:21.589283 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:22.049662 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.049624 2578 generic.go:356] "Generic (PLEG): container finished" podID="a3adff20-0f11-400b-b805-180f82153d5c" containerID="617c045fc81453b5c598cf5e30eb6d45a43295d90a46a529100cb7ffe27639ec" exitCode=0 Jul 08 18:03:22.049877 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.049700 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" event={"ID":"a3adff20-0f11-400b-b805-180f82153d5c","Type":"ContainerDied","Data":"617c045fc81453b5c598cf5e30eb6d45a43295d90a46a529100cb7ffe27639ec"} Jul 08 18:03:22.118545 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.118485 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:22.228101 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228064 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228298 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228124 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228298 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228147 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228298 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228175 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228475 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228475 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228398 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228475 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228453 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228683 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228485 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228683 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4c4d\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d\") pod \"a3adff20-0f11-400b-b805-180f82153d5c\" (UID: \"a3adff20-0f11-400b-b805-180f82153d5c\") " Jul 08 18:03:22.228683 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228573 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:22.228846 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228739 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:22.228846 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228752 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:22.228951 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228905 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:22.229003 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.228989 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.229059 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.229008 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.229059 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.229023 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.229059 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.229037 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3adff20-0f11-400b-b805-180f82153d5c-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.229233 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.229194 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:22.230599 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.230554 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:22.230901 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.230875 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d" (OuterVolumeSpecName: "kube-api-access-s4c4d") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "kube-api-access-s4c4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:22.231083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.231061 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:03:22.231083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.231066 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3adff20-0f11-400b-b805-180f82153d5c" (UID: "a3adff20-0f11-400b-b805-180f82153d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:22.330125 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.330087 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.330125 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.330119 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3adff20-0f11-400b-b805-180f82153d5c-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.330125 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.330128 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3adff20-0f11-400b-b805-180f82153d5c-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.330354 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.330136 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:22.330354 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:22.330146 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4c4d\" (UniqueName: \"kubernetes.io/projected/a3adff20-0f11-400b-b805-180f82153d5c-kube-api-access-s4c4d\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:23.055934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.055903 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" Jul 08 18:03:23.056370 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.055907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh" event={"ID":"a3adff20-0f11-400b-b805-180f82153d5c","Type":"ContainerDied","Data":"d82353d9912b1bb6c072e63effcd87ffd39b4b5df3add85a0786b3e09a129aad"} Jul 08 18:03:23.056370 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.056040 2578 scope.go:117] "RemoveContainer" containerID="617c045fc81453b5c598cf5e30eb6d45a43295d90a46a529100cb7ffe27639ec" Jul 08 18:03:23.074849 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.074818 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:23.081152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.081125 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-141dfad7-openshift-default-6f466b7895-9fzxh"] Jul 08 18:03:23.425533 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:23.425486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:24.309576 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:24.309546 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3adff20-0f11-400b-b805-180f82153d5c" path="/var/lib/kubelet/pods/a3adff20-0f11-400b-b805-180f82153d5c/volumes" Jul 08 18:03:27.885272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:27.885239 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:27.986452 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:27.986417 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:28.039141 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039105 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:28.039728 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039705 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3adff20-0f11-400b-b805-180f82153d5c" containerName="istio-proxy" Jul 08 18:03:28.039814 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039732 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3adff20-0f11-400b-b805-180f82153d5c" containerName="istio-proxy" Jul 08 18:03:28.039814 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039761 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" containerName="maas-api" Jul 08 18:03:28.039814 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039770 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" containerName="maas-api" Jul 08 18:03:28.039977 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039861 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3adff20-0f11-400b-b805-180f82153d5c" containerName="istio-proxy" Jul 08 18:03:28.039977 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.039875 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5115f6e1-5768-4bb0-aaf4-1e1931a8025f" containerName="maas-api" Jul 08 18:03:28.043645 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.043626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.046199 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.046179 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7619a8b2-openshift-default-dockercfg-n4tlp\"" Jul 08 18:03:28.056923 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.056897 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:28.113499 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.113463 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:28.188097 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.187995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188097 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188097 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8bk\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.188388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.188379 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.212902 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.212870 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:28.289346 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289346 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289596 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289596 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8bk\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289596 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289596 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289885 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289885 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289989 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289989 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.289989 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.289980 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.290359 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.290338 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.292207 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.292184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.292411 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.292395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.299441 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.299419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.299673 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.299654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8bk\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk\") pod \"e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.355574 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.355491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:28.495937 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.495907 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:28.498154 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:03:28.498123 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e955721_90d0_4b44_a62a_b31476b677c0.slice/crio-ab5b57f78bffaf690482a408969e813a6301c67861e2c55c891e65ff29c6dae0 WatchSource:0}: Error finding container ab5b57f78bffaf690482a408969e813a6301c67861e2c55c891e65ff29c6dae0: Status 404 returned error can't find the container with id ab5b57f78bffaf690482a408969e813a6301c67861e2c55c891e65ff29c6dae0 Jul 08 18:03:28.500247 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.500214 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:28.500336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.500302 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:28.500336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:28.500329 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:03:29.086616 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.086578 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" event={"ID":"9e955721-90d0-4b44-a62a-b31476b677c0","Type":"ContainerStarted","Data":"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de"} Jul 08 18:03:29.087054 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.086634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" event={"ID":"9e955721-90d0-4b44-a62a-b31476b677c0","Type":"ContainerStarted","Data":"ab5b57f78bffaf690482a408969e813a6301c67861e2c55c891e65ff29c6dae0"} Jul 08 18:03:29.113805 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.113752 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" podStartSLOduration=1.11373583 podStartE2EDuration="1.11373583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:03:29.111914043 +0000 UTC m=+1785.409557212" watchObservedRunningTime="2026-07-08 18:03:29.11373583 +0000 UTC m=+1785.411378998" Jul 08 18:03:29.278325 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.278285 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:29.355907 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.355868 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:29.357291 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.357260 2578 patch_prober.go:28] interesting pod/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 08 18:03:29.357386 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.357308 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 08 18:03:29.377608 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.377566 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:29.477947 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.477918 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:29.480193 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.480175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.482573 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.482551 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7619a8b2-serving-cert\"" Jul 08 18:03:29.492866 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.492831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:29.512716 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.512682 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:29.603588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.603547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6td\" (UniqueName: \"kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.603777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.603664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.603777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.603746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.704874 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.704786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.704874 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.704857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.705065 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.704883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6td\" (UniqueName: \"kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.705065 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:29.704946 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-serving-cert: secret "maas-api-e2e-derive-7619a8b2-serving-cert" not found Jul 08 18:03:29.705065 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:29.705014 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls podName:c0ffa692-7ab9-4a75-8080-85a8e3e05f83 nodeName:}" failed. No retries permitted until 2026-07-08 18:03:30.204998578 +0000 UTC m=+1786.502641723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls") pod "maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" (UID: "c0ffa692-7ab9-4a75-8080-85a8e3e05f83") : secret "maas-api-e2e-derive-7619a8b2-serving-cert" not found Jul 08 18:03:29.705509 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.705488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.716884 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.716863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6td\" (UniqueName: \"kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:29.772187 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.772154 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:29.992701 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:29.992613 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:30.098157 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.098125 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:30.208263 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.208228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:30.210951 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.210927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") pod \"maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:30.360664 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.360636 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:30.392505 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.392468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:30.532736 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:30.532711 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:30.534763 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:03:30.534734 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ffa692_7ab9_4a75_8080_85a8e3e05f83.slice/crio-adb1203371cbb0e563cb283780bac25f08e16c7308fb5c943764de39addf1d3c WatchSource:0}: Error finding container adb1203371cbb0e563cb283780bac25f08e16c7308fb5c943764de39addf1d3c: Status 404 returned error can't find the container with id adb1203371cbb0e563cb283780bac25f08e16c7308fb5c943764de39addf1d3c Jul 08 18:03:31.097171 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:31.097136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" event={"ID":"c0ffa692-7ab9-4a75-8080-85a8e3e05f83","Type":"ContainerStarted","Data":"adb1203371cbb0e563cb283780bac25f08e16c7308fb5c943764de39addf1d3c"} Jul 08 18:03:31.097372 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:31.097343 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:31.098292 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:31.098275 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:32.103497 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:32.103452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" event={"ID":"c0ffa692-7ab9-4a75-8080-85a8e3e05f83","Type":"ContainerStarted","Data":"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575"} Jul 08 18:03:32.103988 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:32.103789 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:32.119997 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:32.119948 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" podStartSLOduration=2.166041827 podStartE2EDuration="3.119931669s" totalImagesPullingTime="953.889842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:03:32.119127741 +0000 UTC m=+1788.416770899" watchObservedRunningTime="2026-07-08 18:03:32.119931669 +0000 UTC m=+1788.417574835" Jul 08 18:03:32.215121 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:32.215086 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:33.655416 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:33.655372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:33.821743 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:33.821658 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:33.844208 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:33.844173 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:34.013551 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.013433 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:34.065963 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.065925 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:34.114473 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.114424 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" containerName="istio-proxy" containerID="cri-o://39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de" gracePeriod=30 Jul 08 18:03:34.114690 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.114545 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" podUID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" containerName="maas-api" containerID="cri-o://d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575" gracePeriod=30 Jul 08 18:03:34.132329 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.132305 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:34.409145 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.409117 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:34.550717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.550632 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca\") pod \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " Jul 08 18:03:34.550717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.550714 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p6td\" (UniqueName: \"kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td\") pod \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " Jul 08 18:03:34.550908 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.550750 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") pod \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\" (UID: \"c0ffa692-7ab9-4a75-8080-85a8e3e05f83\") " Jul 08 18:03:34.551028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.550999 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0ffa692-7ab9-4a75-8080-85a8e3e05f83" (UID: "c0ffa692-7ab9-4a75-8080-85a8e3e05f83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:34.552971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.552945 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0ffa692-7ab9-4a75-8080-85a8e3e05f83" (UID: "c0ffa692-7ab9-4a75-8080-85a8e3e05f83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:03:34.552971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.552945 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td" (OuterVolumeSpecName: "kube-api-access-8p6td") pod "c0ffa692-7ab9-4a75-8080-85a8e3e05f83" (UID: "c0ffa692-7ab9-4a75-8080-85a8e3e05f83"). InnerVolumeSpecName "kube-api-access-8p6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:34.651785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.651749 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-openshift-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:34.651785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.651776 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6td\" (UniqueName: \"kubernetes.io/projected/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-kube-api-access-8p6td\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:34.651785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:34.651786 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0ffa692-7ab9-4a75-8080-85a8e3e05f83-maas-api-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:35.120466 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.120436 2578 generic.go:356] "Generic (PLEG): container finished" podID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" containerID="d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575" exitCode=0 Jul 08 18:03:35.120895 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.120476 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" Jul 08 18:03:35.120895 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.120509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" event={"ID":"c0ffa692-7ab9-4a75-8080-85a8e3e05f83","Type":"ContainerDied","Data":"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575"} Jul 08 18:03:35.120895 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.120573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n" event={"ID":"c0ffa692-7ab9-4a75-8080-85a8e3e05f83","Type":"ContainerDied","Data":"adb1203371cbb0e563cb283780bac25f08e16c7308fb5c943764de39addf1d3c"} Jul 08 18:03:35.120895 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.120595 2578 scope.go:117] "RemoveContainer" containerID="d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575" Jul 08 18:03:35.129205 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.129188 2578 scope.go:117] "RemoveContainer" containerID="d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575" Jul 08 18:03:35.129470 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:35.129452 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575\": container with ID starting with d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575 not found: ID does not exist" containerID="d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575" Jul 08 18:03:35.129549 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.129482 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575"} err="failed to get container status \"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575\": rpc error: code = NotFound desc = could not find container \"d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575\": container with ID starting with d33c0b09930d1b28f734bcb06435f62fb126e0c599cf1ef24c7dd20d8d9ce575 not found: ID does not exist" Jul 08 18:03:35.142996 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.142972 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:35.147800 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.147777 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7619a8b2-74d5d7b65b-dlt2n"] Jul 08 18:03:35.680107 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:35.680078 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:36.310875 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:36.310838 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" path="/var/lib/kubelet/pods/c0ffa692-7ab9-4a75-8080-85a8e3e05f83/volumes" Jul 08 18:03:39.869406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.869382 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:39.903476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903395 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903462 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903556 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903629 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903631 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903670 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n8bk\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.903715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.904050 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903746 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.904050 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903771 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert\") pod \"9e955721-90d0-4b44-a62a-b31476b677c0\" (UID: \"9e955721-90d0-4b44-a62a-b31476b677c0\") " Jul 08 18:03:39.904050 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.903864 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:39.904204 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.904119 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:39.904204 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.904140 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:39.904314 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.904284 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:03:39.904452 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.904410 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:39.904988 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.904961 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:39.905902 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.905879 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:03:39.906353 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.906332 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:39.906469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.906450 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk" (OuterVolumeSpecName: "kube-api-access-4n8bk") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "kube-api-access-4n8bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:03:39.906578 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.906559 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e955721-90d0-4b44-a62a-b31476b677c0" (UID: "9e955721-90d0-4b44-a62a-b31476b677c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:03:39.925210 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:39.925180 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:40.005118 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005083 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005118 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005112 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n8bk\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-kube-api-access-4n8bk\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005118 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005122 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e955721-90d0-4b44-a62a-b31476b677c0-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005352 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005131 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005352 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005139 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e955721-90d0-4b44-a62a-b31476b677c0-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005352 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005148 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e955721-90d0-4b44-a62a-b31476b677c0-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.005352 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.005155 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e955721-90d0-4b44-a62a-b31476b677c0-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:03:40.022963 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.022935 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:40.154401 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.154311 2578 generic.go:356] "Generic (PLEG): container finished" podID="9e955721-90d0-4b44-a62a-b31476b677c0" containerID="39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de" exitCode=0 Jul 08 18:03:40.154401 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.154352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" event={"ID":"9e955721-90d0-4b44-a62a-b31476b677c0","Type":"ContainerDied","Data":"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de"} Jul 08 18:03:40.154401 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.154362 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" Jul 08 18:03:40.154401 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.154394 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v" event={"ID":"9e955721-90d0-4b44-a62a-b31476b677c0","Type":"ContainerDied","Data":"ab5b57f78bffaf690482a408969e813a6301c67861e2c55c891e65ff29c6dae0"} Jul 08 18:03:40.154772 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.154416 2578 scope.go:117] "RemoveContainer" containerID="39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de" Jul 08 18:03:40.163661 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.163641 2578 scope.go:117] "RemoveContainer" containerID="39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de" Jul 08 18:03:40.163914 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:03:40.163899 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de\": container with ID starting with 39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de not found: ID does not exist" containerID="39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de" Jul 08 18:03:40.163965 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.163923 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de"} err="failed to get container status \"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de\": rpc error: code = NotFound desc = could not find container \"39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de\": container with ID starting with 39b7700ac91f66d401d954d308459074a5a8688675856785ff924288ec5224de not found: ID does not exist" Jul 08 18:03:40.178000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.177970 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:40.186065 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.186039 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7619a8b2-openshift-default-5f9b94587f-mqp4v"] Jul 08 18:03:40.309403 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.309362 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" path="/var/lib/kubelet/pods/9e955721-90d0-4b44-a62a-b31476b677c0/volumes" Jul 08 18:03:40.467554 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.467424 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:40.632629 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:40.632595 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:42.118509 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:42.115583 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:48.263677 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:48.263638 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:48.382069 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:48.382034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:48.525764 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:48.525647 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:48.618481 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:48.618451 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:49.159833 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:03:49.159797 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:15.048031 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:15.047992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:20.952724 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:20.952683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:21.039654 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:21.039620 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:21.142993 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:21.142956 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:37.084942 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:37.084905 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:42.972462 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:42.972427 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:43.080418 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:43.080383 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:43.174045 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:43.174010 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:53.722878 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:53.722839 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:55.039621 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:04:55.039573 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.672841 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:00.672809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.778866 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:00.778831 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.875083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:00.875047 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.977970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:00.977882 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:01.883046 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:01.883011 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:02.150266 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:02.150180 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:03.789473 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:03.789389 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:09.683197 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:09.683160 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:09.758991 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:09.758955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:10.437326 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:10.437294 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:16.483444 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:16.483409 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:16.557867 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:16.557830 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:16.752436 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:16.752342 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:17.038721 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.038644 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:17.039161 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039142 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" containerName="istio-proxy" Jul 08 18:05:17.039205 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039163 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" containerName="istio-proxy" Jul 08 18:05:17.039205 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039186 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" containerName="maas-api" Jul 08 18:05:17.039205 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039192 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" containerName="maas-api" Jul 08 18:05:17.039300 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039248 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e955721-90d0-4b44-a62a-b31476b677c0" containerName="istio-proxy" Jul 08 18:05:17.039300 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.039257 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ffa692-7ab9-4a75-8080-85a8e3e05f83" containerName="maas-api" Jul 08 18:05:17.042076 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.042055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.052090 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.052071 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c605a45d-openshift-default-dockercfg-q9b58\"" Jul 08 18:05:17.074431 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.074401 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:17.086967 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.086935 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:17.140316 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140316 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdvz\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.140802 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.140621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.191224 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.191195 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:17.241892 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.241863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.241892 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.241899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.241917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.241942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdvz\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.241994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242416 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242416 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242416 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242562 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242442 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242562 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.242660 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.242589 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.243053 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.243023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.244547 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.244509 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.244867 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.244846 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.251648 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.251623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdvz\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.251727 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.251660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token\") pod \"e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.352794 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.352768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:17.496188 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.496164 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:17.499144 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:05:17.499112 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda741be08_a2d2_4686_95bd_760ff6194879.slice/crio-670b31ba6dd2142a818ecec422eb65b9cbc57a49a47248ba45067b032c451f76 WatchSource:0}: Error finding container 670b31ba6dd2142a818ecec422eb65b9cbc57a49a47248ba45067b032c451f76: Status 404 returned error can't find the container with id 670b31ba6dd2142a818ecec422eb65b9cbc57a49a47248ba45067b032c451f76 Jul 08 18:05:17.501811 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.501776 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:17.501915 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.501846 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:17.501915 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.501876 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:17.627000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.626896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerStarted","Data":"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738"} Jul 08 18:05:17.627000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.626966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerStarted","Data":"670b31ba6dd2142a818ecec422eb65b9cbc57a49a47248ba45067b032c451f76"} Jul 08 18:05:17.655406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:17.652674 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podStartSLOduration=0.65265374 podStartE2EDuration="652.65374ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:05:17.651654115 +0000 UTC m=+1893.949297282" watchObservedRunningTime="2026-07-08 18:05:17.65265374 +0000 UTC m=+1893.950296909" Jul 08 18:05:18.353737 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:18.353705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:18.355334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:18.355309 2578 patch_prober.go:28] interesting pod/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 18:05:18.355438 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:18.355365 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 18:05:19.357178 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:19.357142 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:19.636275 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:19.636187 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:19.637255 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:19.637238 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:20.063314 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.063237 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:20.139407 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.139369 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:20.142705 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.142685 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.145102 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.145077 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c605a45d-serving-cert\"" Jul 08 18:05:20.155441 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.155421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:20.159668 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.159644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:20.236772 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.236732 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:20.271813 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.271773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.272000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.271929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htx5q\" (UniqueName: \"kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.272000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.271980 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.340059 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.339963 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:20.373345 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.373304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htx5q\" (UniqueName: \"kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.373784 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.373371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.373784 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.373436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.374013 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.373993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.376002 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.375986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.384983 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.384951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htx5q\" (UniqueName: \"kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q\") pod \"maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.453589 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.453555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:20.596831 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.596803 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:20.599257 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:05:20.599229 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f448243_1fb1_4294_a19c_469b5595d692.slice/crio-13bae7b0aba2c34fed79e6af746482a57c50538bb4a72bc440740ed884cdefe9 WatchSource:0}: Error finding container 13bae7b0aba2c34fed79e6af746482a57c50538bb4a72bc440740ed884cdefe9: Status 404 returned error can't find the container with id 13bae7b0aba2c34fed79e6af746482a57c50538bb4a72bc440740ed884cdefe9 Jul 08 18:05:20.642147 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.642118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" event={"ID":"4f448243-1fb1-4294-a19c-469b5595d692","Type":"ContainerStarted","Data":"13bae7b0aba2c34fed79e6af746482a57c50538bb4a72bc440740ed884cdefe9"} Jul 08 18:05:20.791451 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.791414 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:20.890286 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:20.890252 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:21.648601 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:21.648560 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" event={"ID":"4f448243-1fb1-4294-a19c-469b5595d692","Type":"ContainerStarted","Data":"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0"} Jul 08 18:05:21.649000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:21.648869 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:21.665560 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:21.665490 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" podStartSLOduration=1.0791287999999999 podStartE2EDuration="1.66547647s" totalImagesPullingTime="586.34767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:05:21.664598465 +0000 UTC m=+1897.962241635" watchObservedRunningTime="2026-07-08 18:05:21.66547647 +0000 UTC m=+1897.963119676" Jul 08 18:05:24.469451 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:24.469414 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:25.196265 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:25.196232 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:25.295488 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:25.295448 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:25.620802 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:25.620770 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:27.659402 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:27.659374 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:31.591279 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:31.591240 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:31.663061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:31.663017 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.003191 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.003135 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.368283 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.368246 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:32.368575 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.368542 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" podUID="4f448243-1fb1-4294-a19c-469b5595d692" containerName="maas-api" containerID="cri-o://fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0" gracePeriod=30 Jul 08 18:05:32.396173 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.396135 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.497444 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.497401 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.620681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.620605 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:32.705853 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.705810 2578 generic.go:356] "Generic (PLEG): container finished" podID="4f448243-1fb1-4294-a19c-469b5595d692" containerID="fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0" exitCode=0 Jul 08 18:05:32.707028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.705859 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" Jul 08 18:05:32.707028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.705891 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" event={"ID":"4f448243-1fb1-4294-a19c-469b5595d692","Type":"ContainerDied","Data":"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0"} Jul 08 18:05:32.707028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.705949 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9" event={"ID":"4f448243-1fb1-4294-a19c-469b5595d692","Type":"ContainerDied","Data":"13bae7b0aba2c34fed79e6af746482a57c50538bb4a72bc440740ed884cdefe9"} Jul 08 18:05:32.707028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.705979 2578 scope.go:117] "RemoveContainer" containerID="fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0" Jul 08 18:05:32.714628 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.714611 2578 scope.go:117] "RemoveContainer" containerID="fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0" Jul 08 18:05:32.714911 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:05:32.714893 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0\": container with ID starting with fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0 not found: ID does not exist" containerID="fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0" Jul 08 18:05:32.714974 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.714920 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0"} err="failed to get container status \"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0\": rpc error: code = NotFound desc = could not find container \"fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0\": container with ID starting with fce4ae806efa240c0c9eed7a7a39910d7ebaafcb646f1d370eacf171bbea0cc0 not found: ID does not exist" Jul 08 18:05:32.794964 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.794929 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htx5q\" (UniqueName: \"kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q\") pod \"4f448243-1fb1-4294-a19c-469b5595d692\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " Jul 08 18:05:32.795152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.794976 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls\") pod \"4f448243-1fb1-4294-a19c-469b5595d692\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " Jul 08 18:05:32.795152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.795105 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca\") pod \"4f448243-1fb1-4294-a19c-469b5595d692\" (UID: \"4f448243-1fb1-4294-a19c-469b5595d692\") " Jul 08 18:05:32.795496 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.795460 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f448243-1fb1-4294-a19c-469b5595d692" (UID: "4f448243-1fb1-4294-a19c-469b5595d692"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:05:32.797298 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.797270 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f448243-1fb1-4294-a19c-469b5595d692" (UID: "4f448243-1fb1-4294-a19c-469b5595d692"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:05:32.797398 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.797344 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q" (OuterVolumeSpecName: "kube-api-access-htx5q") pod "4f448243-1fb1-4294-a19c-469b5595d692" (UID: "4f448243-1fb1-4294-a19c-469b5595d692"). InnerVolumeSpecName "kube-api-access-htx5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:32.895986 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.895905 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f448243-1fb1-4294-a19c-469b5595d692-openshift-service-ca\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:32.895986 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.895940 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htx5q\" (UniqueName: \"kubernetes.io/projected/4f448243-1fb1-4294-a19c-469b5595d692-kube-api-access-htx5q\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:32.895986 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:32.895950 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f448243-1fb1-4294-a19c-469b5595d692-maas-api-tls\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:33.030613 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.030575 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:33.034740 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.034710 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c605a45d-bd79d7c58-qqsj9"] Jul 08 18:05:33.713199 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.713169 2578 generic.go:356] "Generic (PLEG): container finished" podID="a741be08-a2d2-4686-95bd-760ff6194879" containerID="7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738" exitCode=0 Jul 08 18:05:33.713634 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.713205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerDied","Data":"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738"} Jul 08 18:05:33.713679 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.713663 2578 scope.go:117] "RemoveContainer" containerID="7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738" Jul 08 18:05:33.714716 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.714688 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:33.714789 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.714758 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:33.714825 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:33.714813 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:05:34.311405 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:34.311363 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f448243-1fb1-4294-a19c-469b5595d692" path="/var/lib/kubelet/pods/4f448243-1fb1-4294-a19c-469b5595d692/volumes" Jul 08 18:05:34.637149 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:34.637111 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:34.719962 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:34.719925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerStarted","Data":"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769"} Jul 08 18:05:35.353759 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:35.353715 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:35.354984 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:35.354962 2578 patch_prober.go:28] interesting pod/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 18:05:35.355065 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:35.355024 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 18:05:36.353110 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:36.353074 2578 patch_prober.go:28] interesting pod/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 18:05:36.353593 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:36.353138 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 18:05:37.353909 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:37.353875 2578 patch_prober.go:28] interesting pod/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 18:05:37.354301 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:37.353930 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 18:05:38.353918 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.353884 2578 patch_prober.go:28] interesting pod/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 08 18:05:38.354306 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.353940 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 08 18:05:38.642922 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.642828 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:38.752722 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.752683 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:38.930879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.930798 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:38.931022 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.930991 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" containerID="cri-o://c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769" gracePeriod=30 Jul 08 18:05:38.953186 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:38.953155 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:39.347067 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:39.346985 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:39.439699 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:39.439659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:39.610923 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:39.610894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:44.689053 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.689027 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:44.767826 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.767793 2578 generic.go:356] "Generic (PLEG): container finished" podID="a741be08-a2d2-4686-95bd-760ff6194879" containerID="c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769" exitCode=0 Jul 08 18:05:44.767970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.767841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerDied","Data":"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769"} Jul 08 18:05:44.767970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.767848 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" Jul 08 18:05:44.767970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.767894 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf" event={"ID":"a741be08-a2d2-4686-95bd-760ff6194879","Type":"ContainerDied","Data":"670b31ba6dd2142a818ecec422eb65b9cbc57a49a47248ba45067b032c451f76"} Jul 08 18:05:44.767970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.767924 2578 scope.go:117] "RemoveContainer" containerID="c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769" Jul 08 18:05:44.776723 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.776701 2578 scope.go:117] "RemoveContainer" containerID="7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738" Jul 08 18:05:44.784850 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.784829 2578 scope.go:117] "RemoveContainer" containerID="c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769" Jul 08 18:05:44.785139 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:05:44.785118 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769\": container with ID starting with c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769 not found: ID does not exist" containerID="c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769" Jul 08 18:05:44.785273 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.785150 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769"} err="failed to get container status \"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769\": rpc error: code = NotFound desc = could not find container \"c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769\": container with ID starting with c1eff1486acee330a2de6dca6b62a73c71276f9f6c153e0c8b8cadcc71535769 not found: ID does not exist" Jul 08 18:05:44.785273 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.785170 2578 scope.go:117] "RemoveContainer" containerID="7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738" Jul 08 18:05:44.785423 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:05:44.785404 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738\": container with ID starting with 7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738 not found: ID does not exist" containerID="7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738" Jul 08 18:05:44.785467 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.785429 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738"} err="failed to get container status \"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738\": rpc error: code = NotFound desc = could not find container \"7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738\": container with ID starting with 7d3fd43870b6912c34b6f6abcd9a934b7e3f24d1837e48f3299f86402f878738 not found: ID does not exist" Jul 08 18:05:44.807868 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.807839 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.807997 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.807880 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.807997 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.807913 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808072 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.807999 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808072 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808048 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sdvz\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808173 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808090 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808173 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808280 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808170 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:05:44.808280 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808184 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808280 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808220 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data\") pod \"a741be08-a2d2-4686-95bd-760ff6194879\" (UID: \"a741be08-a2d2-4686-95bd-760ff6194879\") " Jul 08 18:05:44.808927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808383 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:05:44.808927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808494 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:05:44.808927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808593 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.808927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808654 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.808927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.808675 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:05:44.809287 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.809031 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data" (OuterVolumeSpecName: "istio-data") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:05:44.810337 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.810315 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token" (OuterVolumeSpecName: "istio-token") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:44.810485 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.810435 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz" (OuterVolumeSpecName: "kube-api-access-6sdvz") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "kube-api-access-6sdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:44.810785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.810759 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:05:44.810785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.810760 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a741be08-a2d2-4686-95bd-760ff6194879" (UID: "a741be08-a2d2-4686-95bd-760ff6194879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:05:44.909625 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909589 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909625 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909619 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909625 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909629 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a741be08-a2d2-4686-95bd-760ff6194879-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909625 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909637 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909645 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a741be08-a2d2-4686-95bd-760ff6194879-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909654 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdvz\" (UniqueName: \"kubernetes.io/projected/a741be08-a2d2-4686-95bd-760ff6194879-kube-api-access-6sdvz\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:44.909900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:44.909662 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a741be08-a2d2-4686-95bd-760ff6194879-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:05:45.089214 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:45.089170 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:45.093266 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:45.093238 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c605a45d-openshift-default-6566f78f9f-nx9nf"] Jul 08 18:05:45.712826 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:45.712772 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:45.871694 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:45.871660 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:45.967887 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:45.967801 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:46.096414 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:46.096377 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:46.315905 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:46.315807 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a741be08-a2d2-4686-95bd-760ff6194879" path="/var/lib/kubelet/pods/a741be08-a2d2-4686-95bd-760ff6194879/volumes" Jul 08 18:05:51.739722 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:51.739685 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:51.840046 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:51.840012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:52.574373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:52.574324 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:52.653893 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:52.653857 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:59.421611 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.421559 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:59.714920 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.714839 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:05:59.715373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715355 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.715419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715377 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.715419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715393 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f448243-1fb1-4294-a19c-469b5595d692" containerName="maas-api" Jul 08 18:05:59.715419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715400 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f448243-1fb1-4294-a19c-469b5595d692" containerName="maas-api" Jul 08 18:05:59.715533 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715423 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.715533 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715432 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.715533 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715509 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f448243-1fb1-4294-a19c-469b5595d692" containerName="maas-api" Jul 08 18:05:59.715634 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715539 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.715634 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.715550 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a741be08-a2d2-4686-95bd-760ff6194879" containerName="istio-proxy" Jul 08 18:05:59.719031 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.719001 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.722717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.722697 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2d05ecf5-openshift-default-dockercfg-5t6v9\"" Jul 08 18:05:59.731990 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.731919 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:05:59.751903 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.751866 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:59.853759 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853844 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74l8x\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.853940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.854176 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.853970 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.854176 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.854005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.890381 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.890330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:59.955184 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74l8x\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955746 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955746 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955746 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955911 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.955911 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.955811 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.956060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.956032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.956275 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.956253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.958144 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.958125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.958536 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.958490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.963408 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.963387 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:05:59.963776 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:05:59.963755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74l8x\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x\") pod \"e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:00.033904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.033793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:00.189033 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.189004 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:06:00.192348 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:06:00.192316 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c76eaf9_344a_430d_9259_659e07d734d6.slice/crio-255682ead02cbeabe4317bf0593004067a8de992c995752f1560cef648bc2099 WatchSource:0}: Error finding container 255682ead02cbeabe4317bf0593004067a8de992c995752f1560cef648bc2099: Status 404 returned error can't find the container with id 255682ead02cbeabe4317bf0593004067a8de992c995752f1560cef648bc2099 Jul 08 18:06:00.194684 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.194647 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:00.194775 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.194718 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:00.194775 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.194750 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:00.846605 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.846567 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerStarted","Data":"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15"} Jul 08 18:06:00.847132 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.846625 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerStarted","Data":"255682ead02cbeabe4317bf0593004067a8de992c995752f1560cef648bc2099"} Jul 08 18:06:00.868742 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:00.868684 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podStartSLOduration=1.868669831 podStartE2EDuration="1.868669831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:06:00.867789433 +0000 UTC m=+1937.165432616" watchObservedRunningTime="2026-07-08 18:06:00.868669831 +0000 UTC m=+1937.166312999" Jul 08 18:06:01.034527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:01.034484 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:01.039498 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:01.039472 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:01.852360 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:01.852334 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:01.853349 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:01.853328 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:02.802193 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:02.802153 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:02.902580 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:02.902541 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:03.000385 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:03.000348 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:03.094838 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:03.094748 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:03.362755 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:03.362718 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:03.464472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:03.464439 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:07.517142 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:07.517108 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:08.319872 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:08.319835 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:17.437974 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.437878 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:55112->10.133.0.46:15021: read: connection reset by peer" start-of-body= Jul 08 18:06:17.438392 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.438347 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:55112->10.133.0.46:15021: read: connection reset by peer" Jul 08 18:06:17.935260 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.935227 2578 generic.go:356] "Generic (PLEG): container finished" podID="9c76eaf9-344a-430d-9259-659e07d734d6" containerID="a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15" exitCode=0 Jul 08 18:06:17.935440 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.935302 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerDied","Data":"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15"} Jul 08 18:06:17.935738 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.935724 2578 scope.go:117] "RemoveContainer" containerID="a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15" Jul 08 18:06:17.936709 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.936680 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:17.936771 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.936750 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:17.936809 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:17.936788 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:06:18.942452 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:18.942420 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerStarted","Data":"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7"} Jul 08 18:06:19.034398 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:19.034350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:19.034649 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:19.034625 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 18:06:19.034698 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:19.034675 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 18:06:19.900613 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:19.900544 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:20.034324 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:20.034287 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 18:06:20.034761 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:20.034344 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 18:06:21.034664 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:21.034621 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 18:06:21.035086 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:21.034683 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 18:06:22.034336 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:22.034298 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 18:06:22.034559 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:22.034355 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 18:06:23.034843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:23.034801 2578 patch_prober.go:28] interesting pod/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 08 18:06:23.035251 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:23.034869 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 08 18:06:24.038443 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:24.038413 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:24.038850 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:24.038827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:24.039636 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:24.039615 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:06:44.938316 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:44.938284 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:45.278926 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:45.278832 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:52.442164 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:52.442128 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:56.047017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:06:56.046984 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:22.735190 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:22.735146 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:22.833872 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:22.833840 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:32.161261 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:32.161223 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:32.850005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:32.849971 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:32.917388 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:32.917348 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:33.442500 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:33.442462 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:39.084891 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:39.084843 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:39.185721 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:39.185688 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:39.400079 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:39.400043 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:07:39.400277 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:39.400255 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" containerID="cri-o://f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7" gracePeriod=30 Jul 08 18:07:39.440936 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:39.440900 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:42.000665 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:42.000625 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:42.186948 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:42.186908 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:44.658956 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.658877 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:07:44.743410 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.743372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:44.767291 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767260 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767489 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767316 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767489 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767357 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767489 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767400 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767489 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767426 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767565 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767597 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:07:44.767744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767625 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74l8x\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767660 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.767744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767705 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs\") pod \"9c76eaf9-344a-430d-9259-659e07d734d6\" (UID: \"9c76eaf9-344a-430d-9259-659e07d734d6\") " Jul 08 18:07:44.768020 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.767749 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:07:44.768091 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.768059 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.768091 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.768076 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.768333 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.768292 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:07:44.768619 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.768562 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:07:44.768784 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.768759 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:07:44.769883 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.769855 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:07:44.770004 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.769988 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:07:44.770267 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.770238 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x" (OuterVolumeSpecName: "kube-api-access-74l8x") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "kube-api-access-74l8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:07:44.770338 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.770321 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c76eaf9-344a-430d-9259-659e07d734d6" (UID: "9c76eaf9-344a-430d-9259-659e07d734d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:07:44.868777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868741 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c76eaf9-344a-430d-9259-659e07d734d6-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.868777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868771 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c76eaf9-344a-430d-9259-659e07d734d6-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.868777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868782 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.869017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868790 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.869017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868799 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74l8x\" (UniqueName: \"kubernetes.io/projected/9c76eaf9-344a-430d-9259-659e07d734d6-kube-api-access-74l8x\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.869017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868809 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:44.869017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:44.868817 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c76eaf9-344a-430d-9259-659e07d734d6-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:07:45.357804 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.357764 2578 generic.go:356] "Generic (PLEG): container finished" podID="9c76eaf9-344a-430d-9259-659e07d734d6" containerID="f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7" exitCode=0 Jul 08 18:07:45.357981 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.357825 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" Jul 08 18:07:45.357981 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.357837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerDied","Data":"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7"} Jul 08 18:07:45.357981 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.357914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t" event={"ID":"9c76eaf9-344a-430d-9259-659e07d734d6","Type":"ContainerDied","Data":"255682ead02cbeabe4317bf0593004067a8de992c995752f1560cef648bc2099"} Jul 08 18:07:45.357981 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.357936 2578 scope.go:117] "RemoveContainer" containerID="f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7" Jul 08 18:07:45.368346 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.368323 2578 scope.go:117] "RemoveContainer" containerID="a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15" Jul 08 18:07:45.377027 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.377004 2578 scope.go:117] "RemoveContainer" containerID="f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7" Jul 08 18:07:45.377351 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:07:45.377323 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7\": container with ID starting with f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7 not found: ID does not exist" containerID="f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7" Jul 08 18:07:45.377424 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.377365 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7"} err="failed to get container status \"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7\": rpc error: code = NotFound desc = could not find container \"f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7\": container with ID starting with f0ed577c8c20f00eb5d01c6b9ca059a82b22b3cb24ad576f2f05e8bb739679f7 not found: ID does not exist" Jul 08 18:07:45.377424 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.377393 2578 scope.go:117] "RemoveContainer" containerID="a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15" Jul 08 18:07:45.377681 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:07:45.377660 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15\": container with ID starting with a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15 not found: ID does not exist" containerID="a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15" Jul 08 18:07:45.377767 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.377683 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15"} err="failed to get container status \"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15\": rpc error: code = NotFound desc = could not find container \"a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15\": container with ID starting with a5f4f1d5f9ab882c9550c76f0fdb91d900d9c22649fb0ed7078197d3a30efb15 not found: ID does not exist" Jul 08 18:07:45.383550 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.383526 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:07:45.389444 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:45.389419 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2d05ecf5-openshift-default-6fb57c544c-qjh9t"] Jul 08 18:07:46.310014 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:46.309983 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" path="/var/lib/kubelet/pods/9c76eaf9-344a-430d-9259-659e07d734d6/volumes" Jul 08 18:07:51.222231 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:51.222196 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:51.301795 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:51.301761 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:51.931090 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:51.931052 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:57.995646 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:57.995603 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:58.097068 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:58.096998 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:58.385732 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:58.385699 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:58.490790 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:07:58.490757 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:09.037196 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:09.037158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:09.400750 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:09.400719 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:30.385258 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:30.385222 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:36.537237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:36.537202 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:36.607510 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:36.607475 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:36.916187 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:36.916158 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:37.262697 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.262604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:08:37.263045 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.263026 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:37.263140 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.263049 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:37.263222 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.263207 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:37.263282 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.263227 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:37.266373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.266351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.270035 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.270012 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a359d5-openshift-default-dockercfg-fj6hg\"" Jul 08 18:08:37.289610 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.289579 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:08:37.296139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.296116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:37.379097 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379296 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25zh\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379296 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379296 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379296 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.379469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.379382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.469736 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.469702 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:37.480907 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.480877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481043 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.480929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481130 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481191 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481247 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481304 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481304 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481304 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481454 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481454 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25zh\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481597 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481561 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481664 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481644 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481720 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481640 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.481720 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.481696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.483658 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.483635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.483843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.483824 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.490775 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.490755 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.491160 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.491132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25zh\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh\") pod \"e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.576321 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.576228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:37.717899 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.717876 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:08:37.720106 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:08:37.720076 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a05df4b_d11a_4432_b9d1_9213ab5aa119.slice/crio-938637e08fe77e3fea4e7bde6364ecae35e43d0a7557ba38f34ae6119c707439 WatchSource:0}: Error finding container 938637e08fe77e3fea4e7bde6364ecae35e43d0a7557ba38f34ae6119c707439: Status 404 returned error can't find the container with id 938637e08fe77e3fea4e7bde6364ecae35e43d0a7557ba38f34ae6119c707439 Jul 08 18:08:37.722294 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.722262 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:37.722404 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.722339 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:37.722404 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:37.722384 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:38.602046 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:38.602004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerStarted","Data":"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96"} Jul 08 18:08:38.602495 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:38.602062 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerStarted","Data":"938637e08fe77e3fea4e7bde6364ecae35e43d0a7557ba38f34ae6119c707439"} Jul 08 18:08:38.625242 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:38.625187 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podStartSLOduration=1.625171703 podStartE2EDuration="1.625171703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:08:38.623425974 +0000 UTC m=+2094.921069178" watchObservedRunningTime="2026-07-08 18:08:38.625171703 +0000 UTC m=+2094.922814869" Jul 08 18:08:39.576399 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:39.576365 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:39.581420 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:39.581395 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:39.606572 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:39.606533 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:39.607458 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:39.607440 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:08:40.615121 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:40.615084 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:40.724436 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:40.724399 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:41.095403 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:41.095312 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:41.329702 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:41.329671 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:41.431904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:41.431866 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:45.156700 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.156668 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:45.524139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.524045 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:45.872641 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.872609 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:08:45.872971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.872960 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:45.873030 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.872973 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c76eaf9-344a-430d-9259-659e07d734d6" containerName="istio-proxy" Jul 08 18:08:45.875727 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.875710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:45.878472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.878453 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c101a9-openshift-default-dockercfg-4xzct\"" Jul 08 18:08:45.894001 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.893972 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:08:45.913463 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:45.913438 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:46.018025 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.017983 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:46.064648 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.064648 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.064879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.064879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.064879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.064879 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.065039 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.065039 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.065039 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.064987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.166399 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.166399 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.166399 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166393 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166852 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.166925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167343 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.167074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.167343 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.167267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.169080 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.169058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.169176 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.169121 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.175999 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.175959 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.176803 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.176782 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token\") pod \"e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.185756 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.185736 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:46.338442 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.338409 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:08:46.341851 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:08:46.341817 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea81006_cd1d_4ce0_ba0d_a1db228eb607.slice/crio-8a8a5552fcb126f0d0095a5d925a2d59de208889a6506ca5cb39969973e636a5 WatchSource:0}: Error finding container 8a8a5552fcb126f0d0095a5d925a2d59de208889a6506ca5cb39969973e636a5: Status 404 returned error can't find the container with id 8a8a5552fcb126f0d0095a5d925a2d59de208889a6506ca5cb39969973e636a5 Jul 08 18:08:46.344273 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.344237 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:46.344373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.344304 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:46.344373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.344333 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:46.640916 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.640877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerStarted","Data":"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe"} Jul 08 18:08:46.641112 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.640930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerStarted","Data":"8a8a5552fcb126f0d0095a5d925a2d59de208889a6506ca5cb39969973e636a5"} Jul 08 18:08:46.666531 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:46.666445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podStartSLOduration=1.666423902 podStartE2EDuration="1.666423902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:08:46.664632357 +0000 UTC m=+2102.962275523" watchObservedRunningTime="2026-07-08 18:08:46.666423902 +0000 UTC m=+2102.964067070" Jul 08 18:08:47.186183 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:47.186152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:47.190932 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:47.190910 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:47.646337 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:47.646300 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:47.647408 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:47.647389 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:08:48.694651 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:48.694616 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:48.796693 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:48.796659 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:48.986995 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:48.986911 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:49.251055 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:49.250967 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:49.506589 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:49.506491 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:49.606138 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:49.606107 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:53.219258 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:53.219220 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:58.710449 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.710413 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerID="c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96" exitCode=0 Jul 08 18:08:58.710915 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.710485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerDied","Data":"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96"} Jul 08 18:08:58.710961 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.710943 2578 scope.go:117] "RemoveContainer" containerID="c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96" Jul 08 18:08:58.712018 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.711988 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:58.712097 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.712079 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:58.712139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:58.712126 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:08:59.720769 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:08:59.720732 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerStarted","Data":"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b"} Jul 08 18:09:00.576435 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:00.576382 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:09:00.576714 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:00.576692 2578 patch_prober.go:28] interesting pod/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 18:09:00.576779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:00.576751 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 18:09:01.577037 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:01.576989 2578 patch_prober.go:28] interesting pod/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 18:09:01.577441 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:01.577067 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 18:09:02.168441 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:02.168403 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:02.268705 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:02.268669 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:02.576708 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:02.576614 2578 patch_prober.go:28] interesting pod/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 18:09:02.576708 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:02.576683 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 18:09:03.576712 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.576660 2578 patch_prober.go:28] interesting pod/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 08 18:09:03.577103 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.576737 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 08 18:09:03.680156 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.679867 2578 patch_prober.go:28] interesting pod/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 18:09:03.680156 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.679933 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 18:09:03.743857 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.743822 2578 generic.go:356] "Generic (PLEG): container finished" podID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerID="ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe" exitCode=0 Jul 08 18:09:03.744063 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.743897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerDied","Data":"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe"} Jul 08 18:09:03.744381 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.744361 2578 scope.go:117] "RemoveContainer" containerID="ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe" Jul 08 18:09:03.745465 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.745435 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:09:03.745585 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.745505 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:09:03.745585 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:03.745556 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:09:04.580189 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:04.580153 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:09:04.580671 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:04.580394 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:09:04.585682 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:04.585649 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:09:04.750237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:04.750202 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerStarted","Data":"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343"} Jul 08 18:09:05.186933 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:05.186894 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:09:05.187319 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:05.187293 2578 patch_prober.go:28] interesting pod/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 18:09:05.187484 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:05.187458 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 18:09:05.967996 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:05.967961 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:06.187033 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:06.186985 2578 patch_prober.go:28] interesting pod/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 18:09:06.187257 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:06.187055 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 18:09:07.186746 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:07.186703 2578 patch_prober.go:28] interesting pod/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 18:09:07.187276 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:07.186779 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 18:09:08.186548 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:08.186480 2578 patch_prober.go:28] interesting pod/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 08 18:09:08.186967 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:08.186935 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 08 18:09:09.189832 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:09.189796 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:09:09.190330 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:09.190172 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:09:09.190916 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:09.190899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:09:10.409531 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:10.409486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:10.512743 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:10.512708 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:30.855765 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.855732 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:09:30.858306 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.858277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.861423 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.861398 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c101a9\"/\"kube-root-ca.crt\"" Jul 08 18:09:30.861582 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.861424 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c101a9\"/\"test-model-c101a9-kserve-self-signed-certs\"" Jul 08 18:09:30.861582 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.861477 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c101a9\"/\"openshift-service-ca.crt\"" Jul 08 18:09:30.861582 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.861425 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c101a9\"/\"default-dockercfg-ccw5s\"" Jul 08 18:09:30.869615 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.869588 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:09:30.896323 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.896287 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:30.909699 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.909853 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.909853 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.909853 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6twp\" (UniqueName: \"kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.909960 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:30.909960 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:30.909897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010420 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010669 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010669 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010669 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6twp\" (UniqueName: \"kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.010843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.010814 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.011065 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.011025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.011129 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.011064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.013078 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.013053 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.013250 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.013233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.022222 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.022181 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6twp\" (UniqueName: \"kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp\") pod \"test-model-c101a9-kserve-7d7b9d895f-8jznv\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.168911 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.168811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:31.307320 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.307234 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:09:31.309865 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:09:31.309838 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4775a6e_f7a5_46b1_9f3c_b8003fc83e4b.slice/crio-3e670007ceb2773d24371d142b24fe156ecabbbbbd54ee830529958c767543e1 WatchSource:0}: Error finding container 3e670007ceb2773d24371d142b24fe156ecabbbbbd54ee830529958c767543e1: Status 404 returned error can't find the container with id 3e670007ceb2773d24371d142b24fe156ecabbbbbd54ee830529958c767543e1 Jul 08 18:09:31.881148 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.881111 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" event={"ID":"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b","Type":"ContainerStarted","Data":"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea"} Jul 08 18:09:31.881551 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.881161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" event={"ID":"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b","Type":"ContainerStarted","Data":"3e670007ceb2773d24371d142b24fe156ecabbbbbd54ee830529958c767543e1"} Jul 08 18:09:31.903533 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:31.903462 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" podStartSLOduration=1.903446308 podStartE2EDuration="1.903446308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:09:31.901554956 +0000 UTC m=+2148.199198124" watchObservedRunningTime="2026-07-08 18:09:31.903446308 +0000 UTC m=+2148.201089476" Jul 08 18:09:41.169715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:41.169605 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:41.169715 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:41.169680 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:41.182364 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:41.182335 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:41.943174 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:41.943146 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:09:57.071671 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:57.071615 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:57.950237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:57.950202 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:09:57.977156 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:57.977125 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:58.139709 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:58.139677 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:58.554211 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:58.554167 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:58.654345 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:58.654306 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:58.859993 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:09:58.859964 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:03.594145 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:03.594109 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:12.970793 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:12.970749 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" podUID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" containerName="main" containerID="cri-o://979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea" gracePeriod=45 Jul 08 18:10:13.220131 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.220103 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:10:13.317164 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317068 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317164 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317112 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317164 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317139 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317164 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317162 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317202 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317232 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6twp\" (UniqueName: \"kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp\") pod \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\" (UID: \"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b\") " Jul 08 18:10:13.317501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317430 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:13.317501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317480 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:13.317742 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317500 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home" (OuterVolumeSpecName: "home") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:13.317742 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317661 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:13.317742 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317681 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:13.317742 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.317695 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:13.319641 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.319622 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm" (OuterVolumeSpecName: "dshm") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:13.319734 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.319633 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:10:13.319734 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.319649 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp" (OuterVolumeSpecName: "kube-api-access-s6twp") pod "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" (UID: "e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b"). InnerVolumeSpecName "kube-api-access-s6twp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:10:13.418361 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.418323 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:13.418361 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.418356 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:13.418627 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:13.418371 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6twp\" (UniqueName: \"kubernetes.io/projected/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b-kube-api-access-s6twp\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:14.085151 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.085113 2578 generic.go:356] "Generic (PLEG): container finished" podID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" containerID="979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea" exitCode=0 Jul 08 18:10:14.085151 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.085151 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" Jul 08 18:10:14.085717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.085171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" event={"ID":"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b","Type":"ContainerDied","Data":"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea"} Jul 08 18:10:14.085717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.085211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv" event={"ID":"e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b","Type":"ContainerDied","Data":"3e670007ceb2773d24371d142b24fe156ecabbbbbd54ee830529958c767543e1"} Jul 08 18:10:14.085717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.085232 2578 scope.go:117] "RemoveContainer" containerID="979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea" Jul 08 18:10:14.095007 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.094988 2578 scope.go:117] "RemoveContainer" containerID="979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea" Jul 08 18:10:14.095298 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:10:14.095278 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea\": container with ID starting with 979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea not found: ID does not exist" containerID="979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea" Jul 08 18:10:14.095375 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.095309 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea"} err="failed to get container status \"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea\": rpc error: code = NotFound desc = could not find container \"979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea\": container with ID starting with 979288c916e602f833c06af188ffb44216641d15b24f4d114fe728c5c00decea not found: ID does not exist" Jul 08 18:10:14.106684 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.106650 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:10:14.112481 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.112452 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c101a9/test-model-c101a9-kserve-7d7b9d895f-8jznv"] Jul 08 18:10:14.311313 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:14.311279 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" path="/var/lib/kubelet/pods/e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b/volumes" Jul 08 18:10:40.877037 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:40.877004 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:41.163789 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:41.163699 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:10:41.163963 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:41.163911 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" containerID="cri-o://27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b" gracePeriod=30 Jul 08 18:10:41.194900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:41.194868 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:41.672338 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:41.672305 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:42.071354 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:42.071265 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:42.241157 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:42.241116 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:42.343410 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:42.343327 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:42.536651 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:42.536614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:46.436396 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.436322 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:10:46.548249 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548215 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548284 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548325 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c25zh\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548348 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548430 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548371 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548438 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548463 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548529 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket\") pod \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\" (UID: \"1a05df4b-d11a-4432-b9d1-9213ab5aa119\") " Jul 08 18:10:46.548779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548625 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:46.548779 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548730 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:46.549032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548836 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:46.549032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548960 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.549032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548975 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.549032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.548988 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.549032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.549005 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:10:46.549431 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.549406 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:46.550767 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.550739 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:10:46.550863 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.550773 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh" (OuterVolumeSpecName: "kube-api-access-c25zh") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "kube-api-access-c25zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:10:46.551406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.551391 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:10:46.551458 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.551434 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a05df4b-d11a-4432-b9d1-9213ab5aa119" (UID: "1a05df4b-d11a-4432-b9d1-9213ab5aa119"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:46.650139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650103 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c25zh\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-kube-api-access-c25zh\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.650139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650133 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.650139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650142 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.650139 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650150 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.650407 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650158 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:46.650407 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:46.650166 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a05df4b-d11a-4432-b9d1-9213ab5aa119-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:47.249282 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.249250 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerID="27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b" exitCode=0 Jul 08 18:10:47.249476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.249294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerDied","Data":"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b"} Jul 08 18:10:47.249476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.249301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" Jul 08 18:10:47.249476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.249335 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k" event={"ID":"1a05df4b-d11a-4432-b9d1-9213ab5aa119","Type":"ContainerDied","Data":"938637e08fe77e3fea4e7bde6364ecae35e43d0a7557ba38f34ae6119c707439"} Jul 08 18:10:47.249476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.249358 2578 scope.go:117] "RemoveContainer" containerID="27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b" Jul 08 18:10:47.258609 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.258582 2578 scope.go:117] "RemoveContainer" containerID="c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96" Jul 08 18:10:47.267305 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.267286 2578 scope.go:117] "RemoveContainer" containerID="27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b" Jul 08 18:10:47.267602 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:10:47.267576 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b\": container with ID starting with 27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b not found: ID does not exist" containerID="27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b" Jul 08 18:10:47.267688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.267618 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b"} err="failed to get container status \"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b\": rpc error: code = NotFound desc = could not find container \"27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b\": container with ID starting with 27bee5e7a2ca70649b53bd6d9c1ef1ef257e76a29ba6064185f9a81ef941545b not found: ID does not exist" Jul 08 18:10:47.267688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.267643 2578 scope.go:117] "RemoveContainer" containerID="c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96" Jul 08 18:10:47.267916 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:10:47.267900 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96\": container with ID starting with c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96 not found: ID does not exist" containerID="c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96" Jul 08 18:10:47.267972 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.267925 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96"} err="failed to get container status \"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96\": rpc error: code = NotFound desc = could not find container \"c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96\": container with ID starting with c7d46713f3360ae26b5e719f6e043166b587716e63fe7bb6f34e9b4b8daddd96 not found: ID does not exist" Jul 08 18:10:47.272582 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.272559 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:10:47.281624 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.279959 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a359d5-openshift-default-694fcfb76c-4gh5k"] Jul 08 18:10:47.985897 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:47.985865 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:48.086550 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.086498 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:48.256709 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.256624 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:10:48.256904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.256882 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" containerID="cri-o://7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343" gracePeriod=30 Jul 08 18:10:48.281152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.281120 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:48.309704 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.309673 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" path="/var/lib/kubelet/pods/1a05df4b-d11a-4432-b9d1-9213ab5aa119/volumes" Jul 08 18:10:48.883837 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:48.883800 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:49.985927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:49.985890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:50.130041 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:50.130008 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:52.612306 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:52.612270 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:52.711405 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:52.711372 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:53.511914 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.511886 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:10:53.617770 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.617739 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.617794 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.617871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.617890 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.617919 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618183 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618212 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618629 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618276 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk\") pod \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\" (UID: \"aea81006-cd1d-4ce0-ba0d-a1db228eb607\") " Jul 08 18:10:53.618629 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618206 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:53.618629 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618348 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:53.618629 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618574 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:10:53.618860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618831 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.618860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618855 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.618969 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618870 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.618969 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.618869 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data" (OuterVolumeSpecName: "istio-data") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:53.619252 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.619096 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:53.620369 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.620342 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:10:53.620369 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.620347 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:10:53.620570 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.620428 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token" (OuterVolumeSpecName: "istio-token") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:10:53.620776 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.620754 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk" (OuterVolumeSpecName: "kube-api-access-qkqfk") pod "aea81006-cd1d-4ce0-ba0d-a1db228eb607" (UID: "aea81006-cd1d-4ce0-ba0d-a1db228eb607"). InnerVolumeSpecName "kube-api-access-qkqfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:10:53.719976 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.719936 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqfk\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-kube-api-access-qkqfk\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.719976 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.719969 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.719976 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.719980 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.719976 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.719987 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.720250 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.719995 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea81006-cd1d-4ce0-ba0d-a1db228eb607-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:53.720250 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:53.720002 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea81006-cd1d-4ce0-ba0d-a1db228eb607-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:10:54.287743 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.287710 2578 generic.go:356] "Generic (PLEG): container finished" podID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerID="7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343" exitCode=0 Jul 08 18:10:54.287954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.287759 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" Jul 08 18:10:54.287954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.287802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerDied","Data":"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343"} Jul 08 18:10:54.287954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.287897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5" event={"ID":"aea81006-cd1d-4ce0-ba0d-a1db228eb607","Type":"ContainerDied","Data":"8a8a5552fcb126f0d0095a5d925a2d59de208889a6506ca5cb39969973e636a5"} Jul 08 18:10:54.287954 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.287927 2578 scope.go:117] "RemoveContainer" containerID="7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343" Jul 08 18:10:54.297316 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.297300 2578 scope.go:117] "RemoveContainer" containerID="ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe" Jul 08 18:10:54.305695 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.305672 2578 scope.go:117] "RemoveContainer" containerID="7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343" Jul 08 18:10:54.305988 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:10:54.305968 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343\": container with ID starting with 7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343 not found: ID does not exist" containerID="7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343" Jul 08 18:10:54.306067 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.306002 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343"} err="failed to get container status \"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343\": rpc error: code = NotFound desc = could not find container \"7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343\": container with ID starting with 7e5de6e6149c8d07b2822e65fc3cdc27cb92e7cf36a683faade406e79565a343 not found: ID does not exist" Jul 08 18:10:54.306067 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.306030 2578 scope.go:117] "RemoveContainer" containerID="ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe" Jul 08 18:10:54.306275 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:10:54.306255 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe\": container with ID starting with ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe not found: ID does not exist" containerID="ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe" Jul 08 18:10:54.306311 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.306282 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe"} err="failed to get container status \"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe\": rpc error: code = NotFound desc = could not find container \"ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe\": container with ID starting with ce75c04fbffdf2ab942f60c9d9f9da257acffa3b56dc025d227f151033cfdefe not found: ID does not exist" Jul 08 18:10:54.311844 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.311821 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:10:54.316211 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.316179 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c101a9-openshift-default-f469b489-w2qg5"] Jul 08 18:10:54.613406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.613369 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:54.889830 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.889735 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:54.988539 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:54.988465 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:55.090991 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:55.090963 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:55.334027 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:55.333925 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:55.434714 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:55.434680 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:56.309627 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:56.309595 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" path="/var/lib/kubelet/pods/aea81006-cd1d-4ce0-ba0d-a1db228eb607/volumes" Jul 08 18:10:59.161032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.160987 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:59.670481 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.670429 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:59.985173 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985098 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:10:59.985462 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985451 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:10:59.985527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985465 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:10:59.985527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985482 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" containerName="main" Jul 08 18:10:59.985527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985487 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" containerName="main" Jul 08 18:10:59.985527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985498 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985527 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985504 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985534 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985540 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985599 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985606 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985613 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a05df4b-d11a-4432-b9d1-9213ab5aa119" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985619 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:10:59.985681 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.985625 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4775a6e-f7a5-46b1-9f3c-b8003fc83e4b" containerName="main" Jul 08 18:10:59.990712 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.990690 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:10:59.994053 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:10:59.994034 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-78cd4b-openshift-default-dockercfg-x44pc\"" Jul 08 18:11:00.008071 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.008047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:11:00.027159 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.027136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:00.080373 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080548 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080548 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080548 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhxn\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080688 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080819 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.080819 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.080796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182322 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhxn\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.182834 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.183272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.183272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.182962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.183272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.183118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.183272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.183223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.184915 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.184897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.185232 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.185216 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.190701 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.190678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.191184 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.191163 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhxn\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn\") pod \"e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.300312 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.300228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:00.450263 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.450236 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:11:00.452553 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:11:00.452526 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2895a4b_f540_4794_b5ed_a7e99efbf83b.slice/crio-336545717a4cc52181da55046fa060e23fc006a4dd993b70d038b097f20feae1 WatchSource:0}: Error finding container 336545717a4cc52181da55046fa060e23fc006a4dd993b70d038b097f20feae1: Status 404 returned error can't find the container with id 336545717a4cc52181da55046fa060e23fc006a4dd993b70d038b097f20feae1 Jul 08 18:11:00.454848 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.454807 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:00.454941 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.454891 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:00.454941 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.454932 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:00.582958 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:00.582875 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:01.325060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:01.325020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerStarted","Data":"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0"} Jul 08 18:11:01.325553 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:01.325074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerStarted","Data":"336545717a4cc52181da55046fa060e23fc006a4dd993b70d038b097f20feae1"} Jul 08 18:11:01.351776 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:01.351730 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podStartSLOduration=2.35171275 podStartE2EDuration="2.35171275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:11:01.350804415 +0000 UTC m=+2237.648447585" watchObservedRunningTime="2026-07-08 18:11:01.35171275 +0000 UTC m=+2237.649355916" Jul 08 18:11:02.300912 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:02.300855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:02.309370 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:02.309346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:02.330360 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:02.330320 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:02.331550 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:02.331507 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:03.303699 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:03.303655 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:03.590254 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:03.590161 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:03.827565 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:03.827461 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:03.961328 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:03.961300 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:04.062372 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:04.062335 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:07.848196 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:07.848151 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:16.908249 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:16.908185 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:17.009154 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:17.009094 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:18.372744 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:18.372702 2578 patch_prober.go:28] interesting pod/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 18:11:18.373143 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:18.372787 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 18:11:19.427808 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.427775 2578 generic.go:356] "Generic (PLEG): container finished" podID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerID="3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0" exitCode=0 Jul 08 18:11:19.428226 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.427849 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerDied","Data":"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0"} Jul 08 18:11:19.428343 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.428314 2578 scope.go:117] "RemoveContainer" containerID="3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0" Jul 08 18:11:19.429490 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.429462 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:19.429581 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.429564 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:19.429648 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:19.429633 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:20.434082 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:20.434044 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerStarted","Data":"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a"} Jul 08 18:11:21.301585 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:21.301472 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:21.301914 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:21.301890 2578 patch_prober.go:28] interesting pod/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 08 18:11:21.302070 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:21.302043 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 08 18:11:22.209937 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.209901 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:22.301475 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.301427 2578 patch_prober.go:28] interesting pod/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 08 18:11:22.301709 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.301536 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 08 18:11:22.332847 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.332805 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:22.490996 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.490894 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:22.988226 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:22.988192 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:23.301487 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.301378 2578 patch_prober.go:28] interesting pod/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 08 18:11:23.301487 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.301452 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 08 18:11:23.334495 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.334464 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:23.459318 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.459280 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:23.576195 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.576098 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:23.790636 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:23.790563 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:24.301257 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:24.301219 2578 patch_prober.go:28] interesting pod/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 08 18:11:24.301617 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:24.301573 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 08 18:11:25.304662 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:25.304625 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:25.305067 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:25.305022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:25.305568 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:25.305550 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:29.679927 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:29.679890 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:29.781305 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:29.781255 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:29.948666 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:29.948576 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:11:29.948816 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:29.948771 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" containerID="cri-o://7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a" gracePeriod=30 Jul 08 18:11:29.964363 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:29.964332 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:30.542534 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:30.542486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:30.877878 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:30.877837 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:31.201944 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:31.201834 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:33.769768 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:33.769738 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:33.868006 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:33.867969 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:33.965300 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:33.965215 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:34.149046 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:34.149012 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:34.350569 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:34.350539 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:34.452992 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:34.452904 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:35.202892 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.202865 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:35.238777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238696 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.238777 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238738 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.238971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238805 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.238971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238866 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.238971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238909 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.238971 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.238939 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjhxn\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.239181 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239098 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.239181 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239149 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.239288 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239199 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo\") pod \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\" (UID: \"b2895a4b-f540-4794-b5ed-a7e99efbf83b\") " Jul 08 18:11:35.239288 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239149 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:11:35.239479 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239409 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:11:35.239479 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239426 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:11:35.239479 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239439 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:11:35.239710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239601 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.239710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239619 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.239710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239632 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.239710 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239645 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.240087 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.239944 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:11:35.241361 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.241332 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn" (OuterVolumeSpecName: "kube-api-access-zjhxn") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "kube-api-access-zjhxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:11:35.241460 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.241369 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:11:35.241910 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.241883 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:11:35.241979 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.241905 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2895a4b-f540-4794-b5ed-a7e99efbf83b" (UID: "b2895a4b-f540-4794-b5ed-a7e99efbf83b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:11:35.340482 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.340446 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.340482 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.340475 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.340482 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.340483 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.340759 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.340494 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhxn\" (UniqueName: \"kubernetes.io/projected/b2895a4b-f540-4794-b5ed-a7e99efbf83b-kube-api-access-zjhxn\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.340759 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.340504 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2895a4b-f540-4794-b5ed-a7e99efbf83b-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:11:35.507406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.507328 2578 generic.go:356] "Generic (PLEG): container finished" podID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerID="7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a" exitCode=0 Jul 08 18:11:35.507406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.507376 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" Jul 08 18:11:35.507619 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.507408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerDied","Data":"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a"} Jul 08 18:11:35.507619 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.507454 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s" event={"ID":"b2895a4b-f540-4794-b5ed-a7e99efbf83b","Type":"ContainerDied","Data":"336545717a4cc52181da55046fa060e23fc006a4dd993b70d038b097f20feae1"} Jul 08 18:11:35.507619 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.507475 2578 scope.go:117] "RemoveContainer" containerID="7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a" Jul 08 18:11:35.517118 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.517100 2578 scope.go:117] "RemoveContainer" containerID="3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0" Jul 08 18:11:35.525156 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.525136 2578 scope.go:117] "RemoveContainer" containerID="7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a" Jul 08 18:11:35.525427 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:11:35.525407 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a\": container with ID starting with 7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a not found: ID does not exist" containerID="7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a" Jul 08 18:11:35.525484 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.525440 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a"} err="failed to get container status \"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a\": rpc error: code = NotFound desc = could not find container \"7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a\": container with ID starting with 7602e1ded91b1a609c59b57f5969ea16d3a6093acb5199c533d4e1bc9b451d5a not found: ID does not exist" Jul 08 18:11:35.525484 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.525458 2578 scope.go:117] "RemoveContainer" containerID="3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0" Jul 08 18:11:35.525751 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:11:35.525731 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0\": container with ID starting with 3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0 not found: ID does not exist" containerID="3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0" Jul 08 18:11:35.525796 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.525756 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0"} err="failed to get container status \"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0\": rpc error: code = NotFound desc = could not find container \"3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0\": container with ID starting with 3af3168061884f5bf56edfb148cd24e2625b906fff9e5754b5431293f555dcc0 not found: ID does not exist" Jul 08 18:11:35.528820 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.528793 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:11:35.531444 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:35.531420 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-78cd4b-openshift-default-7c7c677689-zm87s"] Jul 08 18:11:36.310120 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:36.310088 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" path="/var/lib/kubelet/pods/b2895a4b-f540-4794-b5ed-a7e99efbf83b/volumes" Jul 08 18:11:38.274113 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.274075 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:38.600006 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.599921 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:38.903078 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903037 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:11:38.903390 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903377 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:11:38.903472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903391 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:11:38.903472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903402 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:11:38.903472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903408 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea81006-cd1d-4ce0-ba0d-a1db228eb607" containerName="istio-proxy" Jul 08 18:11:38.903472 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.903472 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:11:38.906272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.906252 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.910701 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.910676 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-63a08f-openshift-default-dockercfg-f7488\"" Jul 08 18:11:38.923824 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.923804 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:11:38.950719 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.950690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:38.975534 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975666 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975666 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975632 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975666 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975804 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975804 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdgx\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975804 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975920 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:38.975920 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:38.975825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076477 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076547 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdgx\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076952 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076952 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.076952 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.076752 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.077196 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.077160 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.077325 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.077208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.077325 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.077233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.077325 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.077275 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.077325 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.077295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.079166 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.079142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.079323 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.079300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.085464 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.085432 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.085737 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.085718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdgx\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx\") pod \"e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.217061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.216964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:39.237268 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.237236 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:39.352328 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.352303 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:11:39.354416 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:11:39.354390 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65e6fb5_0f30_46cd_94a3_860a077023a5.slice/crio-db375de7c89aa26b65b467c2da379092db4a3320b384c7734d8fc250d68c905c WatchSource:0}: Error finding container db375de7c89aa26b65b467c2da379092db4a3320b384c7734d8fc250d68c905c: Status 404 returned error can't find the container with id db375de7c89aa26b65b467c2da379092db4a3320b384c7734d8fc250d68c905c Jul 08 18:11:39.356622 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.356588 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:39.356704 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.356658 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:39.356749 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.356711 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:39.530999 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.530901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerStarted","Data":"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c"} Jul 08 18:11:39.530999 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.530964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerStarted","Data":"db375de7c89aa26b65b467c2da379092db4a3320b384c7734d8fc250d68c905c"} Jul 08 18:11:39.554028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:39.553964 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podStartSLOduration=1.553944598 podStartE2EDuration="1.553944598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:11:39.551183484 +0000 UTC m=+2275.848826651" watchObservedRunningTime="2026-07-08 18:11:39.553944598 +0000 UTC m=+2275.851587768" Jul 08 18:11:40.218028 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:40.217988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:40.223277 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:40.223250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:40.536641 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:40.536556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:40.537696 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:40.537676 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:41.489309 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:41.489277 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:41.590528 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:41.590486 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:41.687747 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:41.687711 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:41.975156 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:41.975123 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:43.388117 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:43.388082 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:43.450846 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:43.450809 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:46.020419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:46.020387 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:52.595958 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.595910 2578 generic.go:356] "Generic (PLEG): container finished" podID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerID="e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c" exitCode=0 Jul 08 18:11:52.596377 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.595964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerDied","Data":"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c"} Jul 08 18:11:52.596420 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.596405 2578 scope.go:117] "RemoveContainer" containerID="e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c" Jul 08 18:11:52.597482 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.597456 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:52.597559 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.597545 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:52.597600 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:52.597578 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 08 18:11:53.606234 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:53.606200 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerStarted","Data":"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56"} Jul 08 18:11:54.217599 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:54.217556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:54.218761 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:54.218739 2578 patch_prober.go:28] interesting pod/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:11:54.218881 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:54.218785 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:11:55.218133 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:55.218088 2578 patch_prober.go:28] interesting pod/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:11:55.218566 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:55.218170 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:11:55.537213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:55.537127 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:56.218304 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:56.218263 2578 patch_prober.go:28] interesting pod/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:11:56.218726 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:56.218332 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:11:56.317633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:56.317602 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:56.415297 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:56.415257 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:57.217843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:57.217800 2578 patch_prober.go:28] interesting pod/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 08 18:11:57.218023 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:57.217865 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 08 18:11:58.222486 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:58.222456 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:11:58.223458 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:11:58.223437 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:12:04.165083 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:04.165046 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:04.270732 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:04.270690 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:05.890172 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:05.890136 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:10.583637 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.583585 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:12:10.584021 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.583957 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:12:10.584021 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.583968 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:12:10.584096 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.584032 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2895a4b-f540-4794-b5ed-a7e99efbf83b" containerName="istio-proxy" Jul 08 18:12:10.593772 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.593751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.596709 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.596686 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"kube-root-ca.crt\"" Jul 08 18:12:10.597564 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.597546 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"auth-test-model-608a65-kserve-self-signed-certs\"" Jul 08 18:12:10.597675 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.597602 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"openshift-service-ca.crt\"" Jul 08 18:12:10.597738 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.597665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"default-dockercfg-jrhdx\"" Jul 08 18:12:10.618641 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.618619 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:12:10.666758 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.666726 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:10.683104 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683057 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.683319 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.683319 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpl2z\" (UniqueName: \"kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.683319 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.683493 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.683493 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.683440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.784897 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.784860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785071 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.784927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785071 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.784967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785071 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785185 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpl2z\" (UniqueName: \"kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785245 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785312 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785293 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785363 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785329 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.785466 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.785451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.787324 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.787306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.787503 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.787485 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.798938 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.798917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpl2z\" (UniqueName: \"kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z\") pod \"auth-test-model-608a65-kserve-5fbf9985d7-bjmzm\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:10.903023 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:10.902986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:11.047850 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:11.047826 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:12:11.049659 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:12:11.049632 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2804eab0_b773_432a_a921_a074d4046abe.slice/crio-a8d1f7f13a02ec3fa43fc81f6383b37baf336b972271ccb0e5b258dcfa390fe5 WatchSource:0}: Error finding container a8d1f7f13a02ec3fa43fc81f6383b37baf336b972271ccb0e5b258dcfa390fe5: Status 404 returned error can't find the container with id a8d1f7f13a02ec3fa43fc81f6383b37baf336b972271ccb0e5b258dcfa390fe5 Jul 08 18:12:11.693285 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:11.693249 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" event={"ID":"2804eab0-b773-432a-a921-a074d4046abe","Type":"ContainerStarted","Data":"6124868558c2885ff630b328130cd57a84e6c1fbd546b565efad2b5787ecd750"} Jul 08 18:12:11.693285 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:11.693293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" event={"ID":"2804eab0-b773-432a-a921-a074d4046abe","Type":"ContainerStarted","Data":"a8d1f7f13a02ec3fa43fc81f6383b37baf336b972271ccb0e5b258dcfa390fe5"} Jul 08 18:12:11.717429 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:11.717381 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" podStartSLOduration=1.717364668 podStartE2EDuration="1.717364668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:12:11.716161456 +0000 UTC m=+2308.013804624" watchObservedRunningTime="2026-07-08 18:12:11.717364668 +0000 UTC m=+2308.015007833" Jul 08 18:12:20.903986 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:20.903949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:20.904419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:20.904006 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:20.916485 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:20.916458 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:21.753682 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:21.753653 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:12:28.475450 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.475410 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:12:28.479015 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.478995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.481548 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.481503 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"openshift-service-ca.crt\"" Jul 08 18:12:28.482547 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.482503 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"kube-root-ca.crt\"" Jul 08 18:12:28.482637 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.482550 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"auth-test-model-63a08f-kserve-self-signed-certs\"" Jul 08 18:12:28.482637 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.482506 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"default-dockercfg-g79qc\"" Jul 08 18:12:28.487309 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.487291 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:12:28.513049 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.513015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:28.557341 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglf6\" (UniqueName: \"kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.557341 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.557635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.557635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.557635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.557635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.557499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.658763 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.658722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglf6\" (UniqueName: \"kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.658763 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.658766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659034 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659100 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659100 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659194 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659433 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659544 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659451 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.659544 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.659502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.661403 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.661369 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.661731 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.661711 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.667760 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.667728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglf6\" (UniqueName: \"kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6\") pod \"auth-test-model-63a08f-kserve-7bf9778c9-rpkp8\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.790580 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.790465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:28.931161 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:28.931115 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:12:28.933249 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:12:28.933222 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a815632_9bb5_4458_be8a_f9169352c205.slice/crio-ab4d81736d57a02d0c7b9bc1335c063be3133bac7896214f2dfcf0e2c05d2a9f WatchSource:0}: Error finding container ab4d81736d57a02d0c7b9bc1335c063be3133bac7896214f2dfcf0e2c05d2a9f: Status 404 returned error can't find the container with id ab4d81736d57a02d0c7b9bc1335c063be3133bac7896214f2dfcf0e2c05d2a9f Jul 08 18:12:29.788021 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:29.787984 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" event={"ID":"0a815632-9bb5-4458-be8a-f9169352c205","Type":"ContainerStarted","Data":"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f"} Jul 08 18:12:29.788021 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:29.788026 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" event={"ID":"0a815632-9bb5-4458-be8a-f9169352c205","Type":"ContainerStarted","Data":"ab4d81736d57a02d0c7b9bc1335c063be3133bac7896214f2dfcf0e2c05d2a9f"} Jul 08 18:12:29.807554 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:29.807479 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" podStartSLOduration=1.8074644100000001 podStartE2EDuration="1.80746441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:12:29.806188143 +0000 UTC m=+2326.103831307" watchObservedRunningTime="2026-07-08 18:12:29.80746441 +0000 UTC m=+2326.105107595" Jul 08 18:12:38.790823 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:38.790727 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:38.790823 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:38.790779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:38.803381 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:38.803350 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:12:38.841039 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:12:38.841009 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:14:32.188147 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:32.188061 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:14:32.237847 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:32.237810 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:33.083255 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:33.083219 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:14:33.097492 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:33.097441 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:39.943301 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:39.943263 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:40.007075 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:40.007045 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:47.208650 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.208604 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" podUID="2804eab0-b773-432a-a921-a074d4046abe" containerName="main" containerID="cri-o://6124868558c2885ff630b328130cd57a84e6c1fbd546b565efad2b5787ecd750" gracePeriod=45 Jul 08 18:14:47.446070 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.446040 2578 generic.go:356] "Generic (PLEG): container finished" podID="2804eab0-b773-432a-a921-a074d4046abe" containerID="6124868558c2885ff630b328130cd57a84e6c1fbd546b565efad2b5787ecd750" exitCode=0 Jul 08 18:14:47.446210 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.446114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" event={"ID":"2804eab0-b773-432a-a921-a074d4046abe","Type":"ContainerDied","Data":"6124868558c2885ff630b328130cd57a84e6c1fbd546b565efad2b5787ecd750"} Jul 08 18:14:47.467129 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.467055 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:14:47.561434 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.561404 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:47.654738 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.654705 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.654940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.654745 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.654940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.654779 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.654940 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.654911 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.655123 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.654994 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.655123 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655029 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home" (OuterVolumeSpecName: "home") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:47.655123 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655053 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpl2z\" (UniqueName: \"kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z\") pod \"2804eab0-b773-432a-a921-a074d4046abe\" (UID: \"2804eab0-b773-432a-a921-a074d4046abe\") " Jul 08 18:14:47.655123 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655060 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache" (OuterVolumeSpecName: "model-cache") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:47.655123 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655107 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:47.655419 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655400 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:47.655457 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655425 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:47.655457 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.655438 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:47.657345 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.657319 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:14:47.657655 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.657621 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm" (OuterVolumeSpecName: "dshm") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:47.657655 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.657630 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z" (OuterVolumeSpecName: "kube-api-access-gpl2z") pod "2804eab0-b773-432a-a921-a074d4046abe" (UID: "2804eab0-b773-432a-a921-a074d4046abe"). InnerVolumeSpecName "kube-api-access-gpl2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:14:47.756278 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.756191 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2804eab0-b773-432a-a921-a074d4046abe-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:47.756278 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.756220 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpl2z\" (UniqueName: \"kubernetes.io/projected/2804eab0-b773-432a-a921-a074d4046abe-kube-api-access-gpl2z\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:47.756278 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:47.756231 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2804eab0-b773-432a-a921-a074d4046abe-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.104628 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.104558 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" podUID="0a815632-9bb5-4458-be8a-f9169352c205" containerName="main" containerID="cri-o://5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f" gracePeriod=45 Jul 08 18:14:48.361176 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.361111 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:14:48.452402 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.452361 2578 generic.go:356] "Generic (PLEG): container finished" podID="0a815632-9bb5-4458-be8a-f9169352c205" containerID="5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f" exitCode=0 Jul 08 18:14:48.452635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.452408 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" Jul 08 18:14:48.452635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.452448 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" event={"ID":"0a815632-9bb5-4458-be8a-f9169352c205","Type":"ContainerDied","Data":"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f"} Jul 08 18:14:48.452635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.452497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8" event={"ID":"0a815632-9bb5-4458-be8a-f9169352c205","Type":"ContainerDied","Data":"ab4d81736d57a02d0c7b9bc1335c063be3133bac7896214f2dfcf0e2c05d2a9f"} Jul 08 18:14:48.452635 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.452549 2578 scope.go:117] "RemoveContainer" containerID="5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f" Jul 08 18:14:48.453746 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.453723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" event={"ID":"2804eab0-b773-432a-a921-a074d4046abe","Type":"ContainerDied","Data":"a8d1f7f13a02ec3fa43fc81f6383b37baf336b972271ccb0e5b258dcfa390fe5"} Jul 08 18:14:48.453845 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.453763 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm" Jul 08 18:14:48.460955 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.460936 2578 scope.go:117] "RemoveContainer" containerID="5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f" Jul 08 18:14:48.461220 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:14:48.461200 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f\": container with ID starting with 5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f not found: ID does not exist" containerID="5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f" Jul 08 18:14:48.461293 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461232 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f"} err="failed to get container status \"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f\": rpc error: code = NotFound desc = could not find container \"5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f\": container with ID starting with 5c745e70e3cea79eb628d267ca4e023f9d0b904f1ff67d332b18c99be81ee89f not found: ID does not exist" Jul 08 18:14:48.461293 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461254 2578 scope.go:117] "RemoveContainer" containerID="6124868558c2885ff630b328130cd57a84e6c1fbd546b565efad2b5787ecd750" Jul 08 18:14:48.461743 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461725 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.461829 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kglf6\" (UniqueName: \"kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.461829 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461805 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.461973 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461833 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.461973 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461858 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.461973 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.461947 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm\") pod \"0a815632-9bb5-4458-be8a-f9169352c205\" (UID: \"0a815632-9bb5-4458-be8a-f9169352c205\") " Jul 08 18:14:48.462229 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.462074 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:48.462229 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.462146 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home" (OuterVolumeSpecName: "home") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:48.462789 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.462342 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.462789 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.462366 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.462984 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.462802 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:48.464374 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.464349 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:14:48.464459 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.464375 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm" (OuterVolumeSpecName: "dshm") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:48.464459 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.464441 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6" (OuterVolumeSpecName: "kube-api-access-kglf6") pod "0a815632-9bb5-4458-be8a-f9169352c205" (UID: "0a815632-9bb5-4458-be8a-f9169352c205"). InnerVolumeSpecName "kube-api-access-kglf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:14:48.475068 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.475041 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:14:48.478901 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.478881 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/auth-test-model-608a65-kserve-5fbf9985d7-bjmzm"] Jul 08 18:14:48.562975 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.562942 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a815632-9bb5-4458-be8a-f9169352c205-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.562975 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.562971 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglf6\" (UniqueName: \"kubernetes.io/projected/0a815632-9bb5-4458-be8a-f9169352c205-kube-api-access-kglf6\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.562975 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.562980 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.563205 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.562989 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a815632-9bb5-4458-be8a-f9169352c205-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:14:48.777272 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.777231 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:14:48.783355 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:48.783330 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/auth-test-model-63a08f-kserve-7bf9778c9-rpkp8"] Jul 08 18:14:50.310161 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.310124 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a815632-9bb5-4458-be8a-f9169352c205" path="/var/lib/kubelet/pods/0a815632-9bb5-4458-be8a-f9169352c205/volumes" Jul 08 18:14:50.310574 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.310545 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2804eab0-b773-432a-a921-a074d4046abe" path="/var/lib/kubelet/pods/2804eab0-b773-432a-a921-a074d4046abe/volumes" Jul 08 18:14:50.358221 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358178 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:14:50.358567 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358552 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a815632-9bb5-4458-be8a-f9169352c205" containerName="main" Jul 08 18:14:50.358567 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358567 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a815632-9bb5-4458-be8a-f9169352c205" containerName="main" Jul 08 18:14:50.358656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358578 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2804eab0-b773-432a-a921-a074d4046abe" containerName="main" Jul 08 18:14:50.358656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358584 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2804eab0-b773-432a-a921-a074d4046abe" containerName="main" Jul 08 18:14:50.358656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358641 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2804eab0-b773-432a-a921-a074d4046abe" containerName="main" Jul 08 18:14:50.358656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.358649 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a815632-9bb5-4458-be8a-f9169352c205" containerName="main" Jul 08 18:14:50.363707 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.363687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.366426 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.366407 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"openshift-service-ca.crt\"" Jul 08 18:14:50.367552 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.367364 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"default-dockercfg-jrhdx\"" Jul 08 18:14:50.367552 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.367384 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"sub-test-model-608a65-kserve-self-signed-certs\"" Jul 08 18:14:50.367552 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.367415 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-608a65\"/\"kube-root-ca.crt\"" Jul 08 18:14:50.372464 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.372442 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:14:50.397839 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.397802 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:50.480627 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480596 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.480798 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lxt\" (UniqueName: \"kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.480798 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.480798 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.480932 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480816 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.480932 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.480856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582375 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582288 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582375 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lxt\" (UniqueName: \"kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582607 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582607 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582607 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582607 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.582897 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.583032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.583032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.582939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.584873 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.584849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.585015 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.584999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.590844 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.590820 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lxt\" (UniqueName: \"kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt\") pod \"sub-test-model-608a65-kserve-cf86b9d47-kwv4q\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.674617 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.674577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:14:50.815874 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:50.815847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:14:50.818048 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:14:50.818024 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fc4923f_860b_447c_ae5b_6ddc7181e9eb.slice/crio-ed0e58ac5b4d0ff88bfb1816ada7296b7412a5e1286cd14557b8e22fe7945c0a WatchSource:0}: Error finding container ed0e58ac5b4d0ff88bfb1816ada7296b7412a5e1286cd14557b8e22fe7945c0a: Status 404 returned error can't find the container with id ed0e58ac5b4d0ff88bfb1816ada7296b7412a5e1286cd14557b8e22fe7945c0a Jul 08 18:14:51.472490 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:51.472456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" event={"ID":"7fc4923f-860b-447c-ae5b-6ddc7181e9eb","Type":"ContainerStarted","Data":"dede71494e7ec6a8e2d897827d41dbb89cea289a2100ca7c64b483b3e21eaeb1"} Jul 08 18:14:51.472490 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:51.472497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" event={"ID":"7fc4923f-860b-447c-ae5b-6ddc7181e9eb","Type":"ContainerStarted","Data":"ed0e58ac5b4d0ff88bfb1816ada7296b7412a5e1286cd14557b8e22fe7945c0a"} Jul 08 18:14:51.489501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:14:51.489445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" podStartSLOduration=1.489424943 podStartE2EDuration="1.489424943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:14:51.488768316 +0000 UTC m=+2467.786411484" watchObservedRunningTime="2026-07-08 18:14:51.489424943 +0000 UTC m=+2467.787068110" Jul 08 18:15:00.674767 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:00.674724 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:15:00.675266 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:00.674783 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:15:00.687665 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:00.687635 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:15:01.532333 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:01.532303 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:15:15.773125 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.773082 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:15:15.781970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.781935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.784695 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.784665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"kube-root-ca.crt\"" Jul 08 18:15:15.784854 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.784667 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"openshift-service-ca.crt\"" Jul 08 18:15:15.784988 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.784969 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:15:15.785560 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.785536 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"sub-test-model-63a08f-kserve-self-signed-certs\"" Jul 08 18:15:15.785754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.785737 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"default-dockercfg-g79qc\"" Jul 08 18:15:15.889106 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.889073 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:15:15.922369 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.922618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.922618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twv5\" (UniqueName: \"kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.922618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.922618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:15.922618 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:15.922550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.023939 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.023844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.023939 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.023888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twv5\" (UniqueName: \"kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.023939 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.023919 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.023939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.023970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.024030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024416 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.024394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.024426 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.024476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.024453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.026353 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.026332 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.026603 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.026583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.032202 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.032175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twv5\" (UniqueName: \"kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5\") pod \"sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.093242 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.093195 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:16.229210 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.229179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:15:16.230641 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:15:16.230613 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a8266ce_bdae_4e38_bbdd_e984f748aff9.slice/crio-a5dad373d749a363135c52c0462db515ebd89233dc965923833536db75854d71 WatchSource:0}: Error finding container a5dad373d749a363135c52c0462db515ebd89233dc965923833536db75854d71: Status 404 returned error can't find the container with id a5dad373d749a363135c52c0462db515ebd89233dc965923833536db75854d71 Jul 08 18:15:16.595936 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.595829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" event={"ID":"1a8266ce-bdae-4e38-bbdd-e984f748aff9","Type":"ContainerStarted","Data":"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890"} Jul 08 18:15:16.595936 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.595882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" event={"ID":"1a8266ce-bdae-4e38-bbdd-e984f748aff9","Type":"ContainerStarted","Data":"a5dad373d749a363135c52c0462db515ebd89233dc965923833536db75854d71"} Jul 08 18:15:16.617573 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:16.617496 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" podStartSLOduration=1.6174722529999999 podStartE2EDuration="1.617472253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:16.615860076 +0000 UTC m=+2492.913503244" watchObservedRunningTime="2026-07-08 18:15:16.617472253 +0000 UTC m=+2492.915115419" Jul 08 18:15:26.093968 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:26.093917 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:26.094549 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:26.093988 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:26.106476 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:26.106445 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:26.666245 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:26.666213 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:15:53.360087 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:53.360049 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:15:53.464406 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:15:53.464373 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:01.667882 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:01.667849 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:16:01.689719 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:01.689687 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:02.582243 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:02.582204 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:16:02.602074 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:02.602041 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:04.238652 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:04.238614 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:13.016239 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:13.016203 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:16.688453 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:16.688407 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" podUID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" containerName="main" containerID="cri-o://dede71494e7ec6a8e2d897827d41dbb89cea289a2100ca7c64b483b3e21eaeb1" gracePeriod=45 Jul 08 18:16:16.920529 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:16.920481 2578 generic.go:356] "Generic (PLEG): container finished" podID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" containerID="dede71494e7ec6a8e2d897827d41dbb89cea289a2100ca7c64b483b3e21eaeb1" exitCode=0 Jul 08 18:16:16.920653 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:16.920547 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" event={"ID":"7fc4923f-860b-447c-ae5b-6ddc7181e9eb","Type":"ContainerDied","Data":"dede71494e7ec6a8e2d897827d41dbb89cea289a2100ca7c64b483b3e21eaeb1"} Jul 08 18:16:16.938828 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:16.938768 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:16:17.023939 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.023907 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024115 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.023963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9lxt\" (UniqueName: \"kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024115 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024115 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024027 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024115 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024102 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024305 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024139 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs\") pod \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\" (UID: \"7fc4923f-860b-447c-ae5b-6ddc7181e9eb\") " Jul 08 18:16:17.024305 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024258 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.024421 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024330 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.024479 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024428 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home" (OuterVolumeSpecName: "home") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.024624 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024605 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.024705 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024628 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.024705 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.024640 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.026307 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.026281 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt" (OuterVolumeSpecName: "kube-api-access-x9lxt") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "kube-api-access-x9lxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:17.026414 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.026370 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm" (OuterVolumeSpecName: "dshm") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.026475 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.026425 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fc4923f-860b-447c-ae5b-6ddc7181e9eb" (UID: "7fc4923f-860b-447c-ae5b-6ddc7181e9eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:16:17.125787 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.125753 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.125787 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.125782 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.126000 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.125795 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lxt\" (UniqueName: \"kubernetes.io/projected/7fc4923f-860b-447c-ae5b-6ddc7181e9eb-kube-api-access-x9lxt\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.603213 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.603132 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" podUID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" containerName="main" containerID="cri-o://ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890" gracePeriod=45 Jul 08 18:16:17.847718 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.847691 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:16:17.926689 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.926600 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" containerID="ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890" exitCode=0 Jul 08 18:16:17.926689 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.926645 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" Jul 08 18:16:17.926904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.926686 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" event={"ID":"1a8266ce-bdae-4e38-bbdd-e984f748aff9","Type":"ContainerDied","Data":"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890"} Jul 08 18:16:17.926904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.926731 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t" event={"ID":"1a8266ce-bdae-4e38-bbdd-e984f748aff9","Type":"ContainerDied","Data":"a5dad373d749a363135c52c0462db515ebd89233dc965923833536db75854d71"} Jul 08 18:16:17.926904 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.926755 2578 scope.go:117] "RemoveContainer" containerID="ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890" Jul 08 18:16:17.927970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.927952 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" event={"ID":"7fc4923f-860b-447c-ae5b-6ddc7181e9eb","Type":"ContainerDied","Data":"ed0e58ac5b4d0ff88bfb1816ada7296b7412a5e1286cd14557b8e22fe7945c0a"} Jul 08 18:16:17.927970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.927971 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q" Jul 08 18:16:17.934537 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934498 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934671 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934557 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934671 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934589 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934792 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934677 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934792 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934713 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7twv5\" (UniqueName: \"kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934906 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934789 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir\") pod \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\" (UID: \"1a8266ce-bdae-4e38-bbdd-e984f748aff9\") " Jul 08 18:16:17.934906 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934848 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home" (OuterVolumeSpecName: "home") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.935022 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.934895 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.935201 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.935153 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.935201 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.935178 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:17.935201 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.935191 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.935784 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.935728 2578 scope.go:117] "RemoveContainer" containerID="ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890" Jul 08 18:16:17.936093 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:16:17.936044 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890\": container with ID starting with ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890 not found: ID does not exist" containerID="ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890" Jul 08 18:16:17.936093 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.936082 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890"} err="failed to get container status \"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890\": rpc error: code = NotFound desc = could not find container \"ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890\": container with ID starting with ae66f1ec025c8474e14e7ebba026bd052134b84ebbdcf4d6fe3f61abd49cb890 not found: ID does not exist" Jul 08 18:16:17.936238 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.936105 2578 scope.go:117] "RemoveContainer" containerID="dede71494e7ec6a8e2d897827d41dbb89cea289a2100ca7c64b483b3e21eaeb1" Jul 08 18:16:17.937165 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.937139 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:16:17.937359 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.937341 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm" (OuterVolumeSpecName: "dshm") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:16:17.937401 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.937379 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5" (OuterVolumeSpecName: "kube-api-access-7twv5") pod "1a8266ce-bdae-4e38-bbdd-e984f748aff9" (UID: "1a8266ce-bdae-4e38-bbdd-e984f748aff9"). InnerVolumeSpecName "kube-api-access-7twv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:17.964104 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.964079 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:16:17.967753 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:17.967725 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-608a65/sub-test-model-608a65-kserve-cf86b9d47-kwv4q"] Jul 08 18:16:18.036314 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.036282 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:18.036314 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.036311 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twv5\" (UniqueName: \"kubernetes.io/projected/1a8266ce-bdae-4e38-bbdd-e984f748aff9-kube-api-access-7twv5\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:18.036314 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.036322 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:18.036545 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.036331 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8266ce-bdae-4e38-bbdd-e984f748aff9-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:16:18.250698 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.250669 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:16:18.255130 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.255108 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/sub-test-model-63a08f-kserve-77b4fdc7ff-8c92t"] Jul 08 18:16:18.311509 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.311471 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" path="/var/lib/kubelet/pods/1a8266ce-bdae-4e38-bbdd-e984f748aff9/volumes" Jul 08 18:16:18.311980 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:18.311959 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" path="/var/lib/kubelet/pods/7fc4923f-860b-447c-ae5b-6ddc7181e9eb/volumes" Jul 08 18:16:29.959459 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.959423 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:16:29.959982 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.959961 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" containerName="main" Jul 08 18:16:29.960061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.959986 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" containerName="main" Jul 08 18:16:29.960061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.960005 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" containerName="main" Jul 08 18:16:29.960061 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.960014 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" containerName="main" Jul 08 18:16:29.960208 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.960122 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8266ce-bdae-4e38-bbdd-e984f748aff9" containerName="main" Jul 08 18:16:29.960208 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.960140 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc4923f-860b-447c-ae5b-6ddc7181e9eb" containerName="main" Jul 08 18:16:29.963348 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.963327 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:29.966005 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.965984 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"openshift-service-ca.crt\"" Jul 08 18:16:29.966116 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.965984 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"kube-root-ca.crt\"" Jul 08 18:16:29.966116 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.965990 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"default-dockercfg-g79qc\"" Jul 08 18:16:29.966450 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.966431 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63a08f\"/\"rate-test-model-63a08f-kserve-self-signed-certs\"" Jul 08 18:16:29.973717 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.973692 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:16:29.992891 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:29.992860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:30.050432 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.050432 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.050711 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.050711 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpfq\" (UniqueName: \"kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.050711 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.050711 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.050701 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151460 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151697 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151697 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151697 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151888 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151693 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpfq\" (UniqueName: \"kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.151888 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.152004 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.152066 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.151998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.152122 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.152066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.153827 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.153797 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.154141 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.154123 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.161055 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.161032 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpfq\" (UniqueName: \"kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq\") pod \"rate-test-model-63a08f-kserve-67d7cf7747-jbqv9\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.273694 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.273602 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:30.412435 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.412402 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:16:30.415532 ip-10-0-134-201 kubenswrapper[2578]: W0708 18:16:30.415481 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10c2b8f3_b80a_4692_b7d1_93d1d1530935.slice/crio-05f2b84387aa50f0c863664a60a66049f90619762e2f66470a6c8adaaf481b00 WatchSource:0}: Error finding container 05f2b84387aa50f0c863664a60a66049f90619762e2f66470a6c8adaaf481b00: Status 404 returned error can't find the container with id 05f2b84387aa50f0c863664a60a66049f90619762e2f66470a6c8adaaf481b00 Jul 08 18:16:30.997583 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.997536 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" event={"ID":"10c2b8f3-b80a-4692-b7d1-93d1d1530935","Type":"ContainerStarted","Data":"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54"} Jul 08 18:16:30.997583 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:30.997588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" event={"ID":"10c2b8f3-b80a-4692-b7d1-93d1d1530935","Type":"ContainerStarted","Data":"05f2b84387aa50f0c863664a60a66049f90619762e2f66470a6c8adaaf481b00"} Jul 08 18:16:31.017501 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:31.017447 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" podStartSLOduration=2.017430926 podStartE2EDuration="2.017430926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:16:31.015937171 +0000 UTC m=+2567.313580339" watchObservedRunningTime="2026-07-08 18:16:31.017430926 +0000 UTC m=+2567.315074092" Jul 08 18:16:40.274634 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:40.274587 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:40.275055 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:40.274658 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:40.287365 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:40.287327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:41.055605 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:41.055571 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:16:57.141997 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:57.141957 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:57.247775 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:16:57.247725 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:04.965443 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:04.965362 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:05.068504 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:05.068472 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:25.361990 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:25.361955 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:26.362402 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:26.362367 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:17:26.372010 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:26.371984 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:31.055125 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:31.055093 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:41.383002 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.382938 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" podUID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" containerName="main" containerID="cri-o://22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54" gracePeriod=45 Jul 08 18:17:41.649970 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.649897 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:17:41.739644 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739609 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.739644 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739658 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.739860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739709 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.739860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739786 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fpfq\" (UniqueName: \"kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.739860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739844 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.740018 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739878 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm\") pod \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\" (UID: \"10c2b8f3-b80a-4692-b7d1-93d1d1530935\") " Jul 08 18:17:41.740018 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.739898 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:41.740018 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.740010 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home" (OuterVolumeSpecName: "home") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:41.740188 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.740033 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:41.740236 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.740201 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tmp-dir\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.740236 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.740227 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-home\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.740302 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.740244 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-model-cache\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.742018 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.741986 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:17:41.742127 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.742013 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm" (OuterVolumeSpecName: "dshm") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:41.742127 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.742025 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq" (OuterVolumeSpecName: "kube-api-access-6fpfq") pod "10c2b8f3-b80a-4692-b7d1-93d1d1530935" (UID: "10c2b8f3-b80a-4692-b7d1-93d1d1530935"). InnerVolumeSpecName "kube-api-access-6fpfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:41.841066 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.841026 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fpfq\" (UniqueName: \"kubernetes.io/projected/10c2b8f3-b80a-4692-b7d1-93d1d1530935-kube-api-access-6fpfq\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.841066 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.841060 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c2b8f3-b80a-4692-b7d1-93d1d1530935-tls-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:41.841066 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:41.841070 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c2b8f3-b80a-4692-b7d1-93d1d1530935-dshm\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:17:42.347707 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.347669 2578 generic.go:356] "Generic (PLEG): container finished" podID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" containerID="22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54" exitCode=0 Jul 08 18:17:42.347894 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.347718 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" Jul 08 18:17:42.347894 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.347735 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" event={"ID":"10c2b8f3-b80a-4692-b7d1-93d1d1530935","Type":"ContainerDied","Data":"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54"} Jul 08 18:17:42.347894 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.347765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9" event={"ID":"10c2b8f3-b80a-4692-b7d1-93d1d1530935","Type":"ContainerDied","Data":"05f2b84387aa50f0c863664a60a66049f90619762e2f66470a6c8adaaf481b00"} Jul 08 18:17:42.347894 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.347781 2578 scope.go:117] "RemoveContainer" containerID="22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54" Jul 08 18:17:42.356333 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.356315 2578 scope.go:117] "RemoveContainer" containerID="22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54" Jul 08 18:17:42.356664 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:17:42.356632 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54\": container with ID starting with 22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54 not found: ID does not exist" containerID="22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54" Jul 08 18:17:42.356719 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.356662 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54"} err="failed to get container status \"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54\": rpc error: code = NotFound desc = could not find container \"22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54\": container with ID starting with 22fe9d6551ae8ea639853a26869a0bcf8bb3d47499dbe207700a9904fb873b54 not found: ID does not exist" Jul 08 18:17:42.364283 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.364262 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:17:42.370147 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:42.370125 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63a08f/rate-test-model-63a08f-kserve-67d7cf7747-jbqv9"] Jul 08 18:17:44.309656 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:44.309622 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" path="/var/lib/kubelet/pods/10c2b8f3-b80a-4692-b7d1-93d1d1530935/volumes" Jul 08 18:17:45.180145 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:17:45.180112 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:08.033052 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:08.033019 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:08.099912 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:08.099878 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:14.180863 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:14.180826 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:14.269437 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:14.269402 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.170917 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.170879 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.268624 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.268591 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.369696 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.369649 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.476121 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.476035 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.576687 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.576654 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:15.773537 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:15.773429 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:21.273357 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:21.273319 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:21.342960 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:21.342920 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:21.530642 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:21.530558 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:18:21.530822 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:21.530782 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" containerID="cri-o://82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56" gracePeriod=30 Jul 08 18:18:21.543169 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:21.543137 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:22.189827 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:22.189776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786cbd6df6-t9p8d_2326706f-69fc-414a-bf1f-b85b1e2f6baf/authorino/0.log" Jul 08 18:18:26.777032 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.777005 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:18:26.874554 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874497 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874596 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874614 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874634 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874654 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874676 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.874754 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.875082 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874787 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.875082 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874830 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grdgx\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx\") pod \"c65e6fb5-0f30-46cd-94a3-860a077023a5\" (UID: \"c65e6fb5-0f30-46cd-94a3-860a077023a5\") " Jul 08 18:18:26.875082 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.874948 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:18:26.875247 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.875154 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:18:26.875247 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.875186 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-credential-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.875372 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.875344 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:18:26.875588 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.875566 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:18:26.876268 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.876244 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:18:26.876881 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.876849 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:18:26.877245 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.877221 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:18:26.877488 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.877462 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:18:26.877488 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.877478 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx" (OuterVolumeSpecName: "kube-api-access-grdgx") pod "c65e6fb5-0f30-46cd-94a3-860a077023a5" (UID: "c65e6fb5-0f30-46cd-94a3-860a077023a5"). InnerVolumeSpecName "kube-api-access-grdgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:18:26.976355 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976321 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-token\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976355 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976348 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-podinfo\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976355 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976359 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c65e6fb5-0f30-46cd-94a3-860a077023a5-istiod-ca-cert\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976368 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-certs\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976375 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-workload-socket\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976384 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-envoy\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976392 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c65e6fb5-0f30-46cd-94a3-860a077023a5-istio-data\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:26.976633 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:26.976399 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grdgx\" (UniqueName: \"kubernetes.io/projected/c65e6fb5-0f30-46cd-94a3-860a077023a5-kube-api-access-grdgx\") on node \"ip-10-0-134-201.ec2.internal\" DevicePath \"\"" Jul 08 18:18:27.129471 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.129388 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67df7499f4-gmf7j_a646702a-4ef7-4f54-aea7-4309b117e277/manager/0.log" Jul 08 18:18:27.490732 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.490648 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-7plc5_86e394c9-4229-4cf1-b75f-98672d79b8d1/copy-manifests/0.log" Jul 08 18:18:27.503131 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.503104 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-7plc5_86e394c9-4229-4cf1-b75f-98672d79b8d1/manager/0.log" Jul 08 18:18:27.566248 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.566213 2578 generic.go:356] "Generic (PLEG): container finished" podID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerID="82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56" exitCode=0 Jul 08 18:18:27.566461 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.566276 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" Jul 08 18:18:27.566461 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.566281 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerDied","Data":"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56"} Jul 08 18:18:27.566461 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.566354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk" event={"ID":"c65e6fb5-0f30-46cd-94a3-860a077023a5","Type":"ContainerDied","Data":"db375de7c89aa26b65b467c2da379092db4a3320b384c7734d8fc250d68c905c"} Jul 08 18:18:27.566461 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.566381 2578 scope.go:117] "RemoveContainer" containerID="82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56" Jul 08 18:18:27.578568 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.578510 2578 scope.go:117] "RemoveContainer" containerID="e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c" Jul 08 18:18:27.586841 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.586816 2578 scope.go:117] "RemoveContainer" containerID="82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56" Jul 08 18:18:27.587198 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:18:27.587169 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56\": container with ID starting with 82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56 not found: ID does not exist" containerID="82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56" Jul 08 18:18:27.587306 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.587242 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56"} err="failed to get container status \"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56\": rpc error: code = NotFound desc = could not find container \"82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56\": container with ID starting with 82914f613f061856699fa809f527d9003ab0bdfb5304db1755001d2176eb1f56 not found: ID does not exist" Jul 08 18:18:27.587306 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.587274 2578 scope.go:117] "RemoveContainer" containerID="e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c" Jul 08 18:18:27.587686 ip-10-0-134-201 kubenswrapper[2578]: E0708 18:18:27.587657 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c\": container with ID starting with e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c not found: ID does not exist" containerID="e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c" Jul 08 18:18:27.587843 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.587695 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c"} err="failed to get container status \"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c\": rpc error: code = NotFound desc = could not find container \"e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c\": container with ID starting with e03ae610a21c5302e5269f84eac7728ec91f942ec6ef7ece7576bbe7c905540c not found: ID does not exist" Jul 08 18:18:27.589491 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.589472 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:18:27.593793 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:27.593771 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-63a08f-openshift-default-8447d9fc99-drnrk"] Jul 08 18:18:28.309631 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:28.309591 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" path="/var/lib/kubelet/pods/c65e6fb5-0f30-46cd-94a3-860a077023a5/volumes" Jul 08 18:18:29.094547 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:29.094502 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786cbd6df6-t9p8d_2326706f-69fc-414a-bf1f-b85b1e2f6baf/authorino/0.log" Jul 08 18:18:29.329676 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:29.329645 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rc9bc_e4e924fd-64bc-4a84-abd1-44b25ad068e8/manager/0.log" Jul 08 18:18:29.450718 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:29.450687 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q9tj6_1ec16576-b065-4b66-b935-592c58ffe0fa/kuadrant-console-plugin/0.log" Jul 08 18:18:30.324583 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:30.324546 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lbwmz_797a9d1b-747d-4441-9d58-fb0d0267e848/istio-proxy/0.log" Jul 08 18:18:30.664945 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:30.664922 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-bxhp7_39631309-4184-4f24-a455-55760b1c12f0/kube-auth-proxy/0.log" Jul 08 18:18:30.788400 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:30.788372 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/1.log" Jul 08 18:18:31.724690 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:31.724664 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-bjbg9_62641589-106e-4c89-b14a-9a810e8128a2/main/0.log" Jul 08 18:18:31.843924 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:31.843898 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwwvqz_52fc0081-c208-403e-a630-ee4e8f10298e/main/0.log" Jul 08 18:18:32.075290 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:32.075212 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-bl9cz_da5180e1-7290-4c71-bc42-665d2c07b86a/main/0.log" Jul 08 18:18:36.666237 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:36.666138 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:36.767079 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:36.767034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" limit=1024 Jul 08 18:18:39.423805 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:39.423775 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8877n_d1921cb0-505b-49ed-b0bc-0c0b0fce8d5f/global-pull-secret-syncer/0.log" Jul 08 18:18:39.538256 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:39.538226 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pcwk6_15225d12-e12f-496b-b822-d7f6f5c666a1/konnectivity-agent/0.log" Jul 08 18:18:39.681469 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:39.681367 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-201.ec2.internal_0779f971e02155c459b2004f758e0fd6/haproxy/0.log" Jul 08 18:18:44.156980 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:44.156949 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786cbd6df6-t9p8d_2326706f-69fc-414a-bf1f-b85b1e2f6baf/authorino/0.log" Jul 08 18:18:44.225700 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:44.225672 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rc9bc_e4e924fd-64bc-4a84-abd1-44b25ad068e8/manager/0.log" Jul 08 18:18:44.258645 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:44.258609 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q9tj6_1ec16576-b065-4b66-b935-592c58ffe0fa/kuadrant-console-plugin/0.log" Jul 08 18:18:46.474497 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.474469 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/alertmanager/0.log" Jul 08 18:18:46.497216 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.497187 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/config-reloader/0.log" Jul 08 18:18:46.520038 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.520007 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/kube-rbac-proxy-web/0.log" Jul 08 18:18:46.544543 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.544478 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/kube-rbac-proxy/0.log" Jul 08 18:18:46.572087 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.572060 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/kube-rbac-proxy-metric/0.log" Jul 08 18:18:46.595952 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.595924 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/prom-label-proxy/0.log" Jul 08 18:18:46.621833 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.621804 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2f1f9ea7-d706-450f-818b-49f3668be1c7/init-config-reloader/0.log" Jul 08 18:18:46.862805 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.862772 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-wj6gc_49b64cca-249a-4df2-bc5e-cc33a2d57b86/monitoring-plugin/0.log" Jul 08 18:18:46.990577 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:46.990543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zpjm_a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8/node-exporter/0.log" Jul 08 18:18:47.018214 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.018184 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zpjm_a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8/kube-rbac-proxy/0.log" Jul 08 18:18:47.050150 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.050123 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6zpjm_a7a71e8d-af6c-4dd9-8b57-d15e7330a1d8/init-textfile/0.log" Jul 08 18:18:47.524152 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.524118 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/prometheus/0.log" Jul 08 18:18:47.548574 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.548496 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/config-reloader/0.log" Jul 08 18:18:47.571678 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.571643 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/thanos-sidecar/0.log" Jul 08 18:18:47.595360 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.595327 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/kube-rbac-proxy-web/0.log" Jul 08 18:18:47.618460 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.618428 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/kube-rbac-proxy/0.log" Jul 08 18:18:47.643529 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.643495 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/kube-rbac-proxy-thanos/0.log" Jul 08 18:18:47.671135 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.671042 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d1b9d204-def9-4356-80f4-cbd8a0606f99/init-config-reloader/0.log" Jul 08 18:18:47.701759 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.701699 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-946gp_b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c/prometheus-operator/0.log" Jul 08 18:18:47.723355 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:47.723329 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-946gp_b05fcc7a-9ba8-4f3d-8404-3ead2dcded3c/kube-rbac-proxy/0.log" Jul 08 18:18:49.850546 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850492 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn"] Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850860 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" containerName="main" Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850872 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" containerName="main" Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850887 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850893 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850902 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.850934 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850907 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.851142 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850961 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.851142 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850969 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c2b8f3-b80a-4692-b7d1-93d1d1530935" containerName="main" Jul 08 18:18:49.851142 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.850975 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65e6fb5-0f30-46cd-94a3-860a077023a5" containerName="istio-proxy" Jul 08 18:18:49.853669 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.853643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:49.856188 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.856164 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq44k\"/\"kube-root-ca.crt\"" Jul 08 18:18:49.857207 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.857187 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cq44k\"/\"openshift-service-ca.crt\"" Jul 08 18:18:49.857312 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.857196 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cq44k\"/\"default-dockercfg-5jf2t\"" Jul 08 18:18:49.866594 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.866565 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn"] Jul 08 18:18:49.997417 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.997375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-sys\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:49.997417 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.997423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5x2\" (UniqueName: \"kubernetes.io/projected/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-kube-api-access-9p5x2\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:49.997665 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.997451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-lib-modules\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:49.997665 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.997568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-proc\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:49.997665 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:49.997607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-podres\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098761 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-proc\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-podres\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-sys\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5x2\" (UniqueName: \"kubernetes.io/projected/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-kube-api-access-9p5x2\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098855 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-proc\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.098900 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-lib-modules\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.099060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-sys\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.099060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.098983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-podres\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.099060 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.099042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-lib-modules\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.106707 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.106683 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5x2\" (UniqueName: \"kubernetes.io/projected/5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7-kube-api-access-9p5x2\") pod \"perf-node-gather-daemonset-p7cwn\" (UID: \"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7\") " pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.164785 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.164751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.305165 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.301951 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn"] Jul 08 18:18:50.424070 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.424044 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68f66b4b69-9fthx_01844b12-7984-4ed5-8779-ee3b94d38e18/console/0.log" Jul 08 18:18:50.680027 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.679926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" event={"ID":"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7","Type":"ContainerStarted","Data":"f6a4331cfd6da445773f8d3e4ff7b3950cf6e0f7d09c589f0f2d600359c4bd2a"} Jul 08 18:18:50.680027 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.679979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" event={"ID":"5a9a66a8-60a2-48b3-a5c3-c4e8aa3907f7","Type":"ContainerStarted","Data":"cac6afe78bde50e379ab203f951fa8e1e2f06ae762c6306976a655c402bef642"} Jul 08 18:18:50.680256 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.680026 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:50.698106 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:50.698052 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" podStartSLOduration=1.698038919 podStartE2EDuration="1.698038919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:50.697187652 +0000 UTC m=+2706.994830820" watchObservedRunningTime="2026-07-08 18:18:50.698038919 +0000 UTC m=+2706.995682085" Jul 08 18:18:52.139853 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:52.139824 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gv4qs_7235a83b-fd41-46fe-a258-557f127f8b08/dns/0.log" Jul 08 18:18:52.164375 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:52.164350 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gv4qs_7235a83b-fd41-46fe-a258-557f127f8b08/kube-rbac-proxy/0.log" Jul 08 18:18:52.239733 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:52.239707 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9x9pr_695e5fbc-e134-4402-b44c-2780e00cdb77/dns-node-resolver/0.log" Jul 08 18:18:52.883130 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:52.883097 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nllq2_58ad5d1d-145d-40b4-abc7-bb7496a78f9a/node-ca/0.log" Jul 08 18:18:53.921286 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:53.921257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lbwmz_797a9d1b-747d-4441-9d58-fb0d0267e848/istio-proxy/0.log" Jul 08 18:18:54.107866 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:54.107839 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-bxhp7_39631309-4184-4f24-a455-55760b1c12f0/kube-auth-proxy/0.log" Jul 08 18:18:54.139968 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:54.139935 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/0.log" Jul 08 18:18:54.165361 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:54.165335 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/1.log" Jul 08 18:18:54.972263 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:54.972232 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-679nq_b02a4077-3343-4838-b39a-04e554934119/serve-healthcheck-canary/0.log" Jul 08 18:18:55.666151 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:55.666119 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxntr_6cec3255-c25a-4c5e-ae32-65e5f5126624/kube-rbac-proxy/0.log" Jul 08 18:18:55.687478 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:55.687450 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxntr_6cec3255-c25a-4c5e-ae32-65e5f5126624/exporter/0.log" Jul 08 18:18:55.709755 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:55.709722 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xxntr_6cec3255-c25a-4c5e-ae32-65e5f5126624/extractor/0.log" Jul 08 18:18:56.695561 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:56.695509 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cq44k/perf-node-gather-daemonset-p7cwn" Jul 08 18:18:58.033041 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:58.033003 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-67df7499f4-gmf7j_a646702a-4ef7-4f54-aea7-4309b117e277/manager/0.log" Jul 08 18:18:58.189421 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:58.189389 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-7plc5_86e394c9-4229-4cf1-b75f-98672d79b8d1/manager/0.log" Jul 08 18:18:58.209888 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:58.209861 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c548d5447-7plc5_86e394c9-4229-4cf1-b75f-98672d79b8d1/copy-manifests/0.log" Jul 08 18:18:59.711317 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:59.711288 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8d6bb58cc-gg69n_dcf13efa-bc07-4270-b5eb-ea81203fa514/manager/0.log" Jul 08 18:18:59.764300 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:18:59.764270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xjpzs_521be8a3-2681-473c-9140-cbadba624596/openshift-lws-operator/0.log" Jul 08 18:19:00.141056 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:00.141017 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lbwmz_797a9d1b-747d-4441-9d58-fb0d0267e848/istio-proxy/0.log" Jul 08 18:19:00.302299 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:00.302268 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-bxhp7_39631309-4184-4f24-a455-55760b1c12f0/kube-auth-proxy/0.log" Jul 08 18:19:00.320446 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:00.320400 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/0.log" Jul 08 18:19:00.348876 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:00.348839 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/1.log" Jul 08 18:19:01.598274 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:01.598197 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4lbwmz_797a9d1b-747d-4441-9d58-fb0d0267e848/istio-proxy/0.log" Jul 08 18:19:01.774714 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:01.774680 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6674766f74-bxhp7_39631309-4184-4f24-a455-55760b1c12f0/kube-auth-proxy/0.log" Jul 08 18:19:01.793819 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:01.793790 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/0.log" Jul 08 18:19:01.821334 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:01.821299 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-p7t6f_7bd20f55-ae1e-4ab1-8848-4b001f3c4883/istio-proxy/1.log" Jul 08 18:19:06.602017 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.601986 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/kube-multus-additional-cni-plugins/0.log" Jul 08 18:19:06.624384 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.624359 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/egress-router-binary-copy/0.log" Jul 08 18:19:06.648575 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.648544 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/cni-plugins/0.log" Jul 08 18:19:06.670573 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.670505 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/bond-cni-plugin/0.log" Jul 08 18:19:06.692881 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.692855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/routeoverride-cni/0.log" Jul 08 18:19:06.715711 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.715677 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/whereabouts-cni-bincopy/0.log" Jul 08 18:19:06.738199 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.738159 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bjrww_bd3e0e2d-0cad-4849-9a61-3c03d68d1fb9/whereabouts-cni/0.log" Jul 08 18:19:06.932860 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:06.932759 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h9rp6_c0808e76-6d3d-42da-8ba4-b39654a015b3/kube-multus/0.log" Jul 08 18:19:07.023630 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:07.023583 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g4jl4_71fef467-1e6d-45d5-b3fb-78ede7b5caff/network-metrics-daemon/0.log" Jul 08 18:19:07.045081 ip-10-0-134-201 kubenswrapper[2578]: I0708 18:19:07.045053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g4jl4_71fef467-1e6d-45d5-b3fb-78ede7b5caff/kube-rbac-proxy/0.log"